Systems and methods for enhanced imaging and analysis

ABSTRACT

A method to, is provided for collecting an image from a sample. The method includes selecting a radiation level for a first probe to meet a desired radiation dosage, and providing, with the first probe, a radiation at a selected point within a region of the sample. The method includes identifying a second selected point within the region of the sample based on a down sampling scheme, and providing a second radiation amount at the second selected point within the region of the sample. The method also includes interpolating a first datum and a second datum based on an up sampling scheme to obtain a plurality of data, and forming an image of the region of the sample with the plurality of data. A system to perform the above method and including the first probe is also provided.

ACKNOWLEDGMENT OF GOVERNMENT SUPPORT

This invention was made with government support under NIH CA014195awarded by the National Institutes of Health. The government has certainrights in the invention.

BACKGROUND Field

The embodiments disclosed herein are generally directed towards systemsand methods for enhancing imaging analysis. More specifically,embodiments disclosed herein are directed to the use of non-linear imageprocessing algorithms to produce enhanced images on the back end of animaging system, thus relaxing the constraints for the image collectionhardware and procedures.

Description of Related Art

Current imaging instrumentation has evolved substantially with theavailability of powerful devices, e.g., lasers, electron beams, andother high-resolution, high-energy radiation sources (e.g., X-rays,isotope radiation) and more sensitive detectors having multi-channeldetection capabilities. In many instances, the acquisition of thehigh-resolution images that such devices allow involves long scanningtimes and exposure of the subject sample to potential radiation damageand other undesirable alterations (drift, heating, and temperaturegradients). Additionally, implementing high-resolution imaging at thefront end of the instrumentation (e.g., via hardware for probing anddetection) typically reduces the time resolution of the imagingprotocol, compared to a lower resolution image obtained with the samehardware (e.g., at a coarser scanning rate). It is desirable to applyimage-enhancing techniques on the back end of the imaging system torelax the interaction between probe and detector with the sample, whilemaintaining or improving the image quality.

SUMMARY

In a first embodiment, a method for collecting an image from a sampleincludes selecting a radiation level for a first probe to meet a desiredradiation dosage, and providing, with the first probe, a first radiationamount at a first selected point within a region of the sample, based onthe radiation level. The method also includes associating the firstselected point with at least the portion of a first emitted radiationresulting from an interaction of the first radiation amount with thesample, to form a first datum, identifying a second selected pointwithin the region of the sample based on a down sampling scheme, andproviding, with the first probe, a second radiation amount at the secondselected point within the region of the sample. The method also includesassociating the second selected point with at least the portion of asecond emitted radiation resulting from the interaction of the secondradiation amount with the sample, to form a second datum andinterpolating the first datum and the second datum based on an upsampling scheme to obtain at least a third datum. The method alsoincludes obtaining a plurality of data from multiple selected points ina portion of the region of the sample and forming an image of the regionof the sample with the plurality of data.

In a second embodiment, a system for collecting an image from a sampleincludes a first probe configured to deliver a radiation to a selectedpoint in the sample, and a first detector configured to measure ascattered radiation resulting from an interaction between the radiationand the sample. The system also includes a memory storing instructionsand one or more processors configured to execute the instructions and tocause the system to select a radiation level for a first probe to meet adesired radiation dosage. The one or more processors also cause thesystem to provide, with the first probe, a first radiation amount at afirst selected point within a region of the sample, based on theradiation level, and to associate the first selected point with at leastthe portion of a first emitted radiation resulting from an interactionof the first radiation amount with the sample, to form a first datum.The one or more processors also cause the system to identify a secondselected point within the region of the sample based on a down samplingscheme, to provide, with a first probe, a second radiation amount at thesecond selected point within the region of the sample and to associatethe second selected point with at least the portion of a second emittedradiation resulting from the interaction of the second radiation amountwith the sample, to form a second datum. The one or more processors alsocause the system to interpolate the first datum and the second datumbased on an up sampling scheme to obtain at least a third datum, toobtain a plurality of data from multiple selected points in a portion ofthe region of the sample, and to form an image of the region of thesample with the plurality of data.

In yet other embodiment, a computer-implemented method to train analgorithm for collecting an image of a sample includes retrieving ahigh-resolution image of a known sample and identifying a firstclassifier for the high-resolution image of the known sample, whereinthe first classifier includes a first value. The computer-implementedmethod also includes aggregating, with a selected coefficient, one ormore pixels in the high-resolution image to obtain a low-resolutionimage of the sample, wherein the one or more pixels are selected basedon a desired down sampling of an image collection system, and obtaininga second classifier for the low-resolution image of the sample, whereinthe second classifier includes a second value. The computer-implementedmethod also includes determining a metric value with a differencebetween the second value and the first value and modifying the selectedcoefficient.

BRIEF DESCRIPTION OF FIGURES

FIG. 1 illustrates an architecture including imaging instrumentation andassociated control devices for enhanced imaging, according to variousembodiments.

FIG. 2 illustrates a detailed block diagram of a system in thearchitecture of FIG. 1, according to various embodiments.

FIG. 3 illustrates a block diagram of a convolutional neural network forenhanced imaging, according to various embodiments.

FIG. 4 illustrates a detailed block diagram of a convolutional neuralnetwork for enhanced imaging, according to various embodiments.

FIG. 5 illustrates a block diagram of a generative adversarial networkfor enhanced imaging, according to various embodiments.

FIGS. 6A and 6B illustrate an image enhancement of a neuron obtained,according to various embodiments.

FIGS. 7A through 7D illustrate image enhancements on a portion of theimage in FIGS. 6A-6B obtained, according to various embodiments.

FIGS. 8A and 8B illustrate an image enhancement of a mitochondriaobtained, according to various embodiments.

FIGS. 9A through 9D illustrate an image enhancement of a portion of theimage in FIGS. 8A-8B obtained, according to various embodiments.

FIGS. 10A through 10C illustrate an image enhancement of a sub-Nyquistunder sampled image obtained with a scanning electron microscope,according to various embodiments.

FIG. 11 illustrates a detailed block diagram for a semi-synthetictraining scheme in a neural network for enhanced imaging, according tovarious embodiments.

FIGS. 12A through 12C illustrate some applications of the neural networkin FIG. 11 for electron microscopy, according to various embodiments.

FIGS. 13A through 13E illustrate some applications of the neural networkin FIG. 11 for electron microscopy, according to various embodiments.

FIGS. 14A through 14K illustrate some applications of the neural networkin confocal fluorescence microscopy, according to various embodiments.

FIG. 15 illustrates steps in a method for controlling an imaginginstrumentation to obtain an enhanced image, according to variousembodiments.

FIG. 16 illustrates steps in a method for training an algorithm tocontrol imaging instrumentation to obtain an enhanced image, accordingto various embodiments.

FIG. 17 illustrates a block diagram of a system configured to obtain anenhanced image from an imaging instrumentation, according to variousembodiments.

It is to be understood that the figures are not necessarily drawn toscale, nor are the objects in the figures necessarily drawn to scale inrelationship to one another. The figures are depictions that areintended to bring clarity and understanding to various embodiments ofapparatuses, systems, and methods disclosed herein. Wherever possible,the same reference numbers will be used throughout the drawings to referto the same or like parts. Moreover, it should be appreciated that thedrawings are not intended to limit the scope of the present teachings inany way.

DETAILED DESCRIPTION

This specification describes various exemplary embodiments of systems,methods, and software for enhanced novelty detection. The disclosure,however, is not limited to these exemplary embodiments and applicationsor to the manner in which the exemplary embodiments and applicationsoperate or are described herein.

Unless otherwise defined, scientific and technical terms used inconnection with the present teachings described herein shall have themeanings that are commonly understood by those of ordinary skill in theart. Further, unless otherwise required by context, singular terms shallinclude pluralities and plural terms shall include the singular.

As used herein, the terms “comprise,” “comprises,” “comprising,”“contain,” “contains,” “containing,” “have,” “having,” “include,”“includes,” and “including” and their variants are not intended to belimiting, are inclusive or open-ended and do not exclude additional,unrecited additives, components, integers, elements, or method steps.For example, a process, method, system, composition, kit, or apparatusthat includes a list of features is not necessarily limited only tothose features but may include other features not expressly listed orinherent to such process, method, system, composition, kit, orapparatus.

In accordance with various embodiments herein, the systems, methods, andsoftware are described for high-resolution cellular and tissue imaging,including point scanning imaging systems combined with microscopy. Pointscanning imaging systems in embodiments consistent with the presentdisclosure may include a scanning electron microscope (SEM) or ascanning transmission electron microscope (STEM), an ion-based imagingsystem or a high-resolution cryoSTEM. In some embodiments, a pointscanning imaging system includes a laser scanning confocal microscope,or a fluorescence microscope, and the like.

For imaging systems, it is desirable to optimize resolution, speed,illumination intensity, and signal-to-noise ratio (SNR). For pointscanning systems, optimizing all of the above simultaneously is achallenging task. State-of-the-art systems typically incur inundesirable compromises between the above factors. In someconfigurations, point scanning systems are constrained by the fact thathigher resolution imaging requires a higher number of pixels for propersampling, resulting in a direct relationship between imaging time andpixel resolution. In addition, the increased imaging time results in ahigher dose of radiation transferred to the sample, for imaging. Thehigher dose may be deleterious to the measure from direct damage to thesample (e.g., alter the physiology or the nature of the sample) and fromthermal drifts and heating that may result from the radiation.

Similar to SEM, laser scanning confocal microscopy also suffers from adirect relationship between pixel resolution and sample damage (e.g.,phototoxicity/photobleaching). This can be a major barrier for cellbiologists who wish to study the dynamics of smaller structures such asmitochondria, which regularly undergo fission and fusion, but also showincreased fission and swelling in response to phototoxicity. In extremecases, phototoxicity can cause cell death, which is incompatible withlive cell imaging (data not shown). HR scanning confocal microscopy alsosuffers from the direct relationship between pixel resolution andimaging time, making live cell imaging of faster processes challenging(e.g., organelle motility in neurons).

Embodiments as disclosed herein mitigate the challenges described aboveby combining a low-resolution hardware approach (e.g., less invasive,less costly, and faster) with an enhanced data processing software torecover the desired high resolution. In some embodiments, the solutionsdisclosed herein can overcome the above limitations by implementing asolution in the field of computer technology, namely the use ofpoint-scanning super-resolution (PSSR) models to compensate for thehardware limitations of an imaging system. In some embodiments, PSSR isimplemented with deep-learning algorithms, neural network algorithms, orartificial intelligence algorithms that are trained against a largestock of previously collected and stored high-resolution images. In someembodiments, the high-resolution images are oversampled, “ground truth,”images acquired on scanning transmission electron (STEM) or laserscanning confocal microscopes. In some embodiments, high-resolutionimages are used to generate semi-synthetic training data for creatingPSSR models. The PSSR models are then used to restore (e.g., upgrade orenhance) under sampled images. In some embodiments, the PSSR modelsdisclosed herein may restore under sampled images acquired withdifferent optics, detectors, samples, or sample preparation methods,relative to the training data sets.

The ability to restore or enhance under sampled images enables theacquisition of previously unattainable resolution and SNR datasets withstandard scanning imaging systems. This broadens the range andcapabilities of the scanning imaging systems into new and challengingapplications. Accordingly, PSSR models as disclosed herein provide ahighly practical and versatile strategy for point scanning imaging withoptimal resolution, speed, and sensitivity.

For example, in the case of a laser scanning confocal fluorescencemicroscope, the higher speeds enabled by under sampled images combinedwith PSSR models as disclosed herein facilitates otherwise unattainablespatiotemporal resolution. This may be used to study physiologicalphenomena such as mitochondrial dynamics for prolonged periods of timewith reduced phototoxicity or photobleaching. In addition tophototoxicity issues, the slow speed of HR scanning confocal imagingresults in temporal under sampling of fast-moving structures such asmotile mitochondria in neurons. Accordingly, in some embodiments, a PSSRmodel as disclosed herein may provide sufficient restoration of undersampled time-lapse-imaging of mitochondrial trafficking in neurons, orfission or fusion events.

In some embodiments, PSSR models as disclosed herein may be used forimage analysis and segmentation by using segmentation atlases or mapstrained into the PSSR model before image collection. For example, insome embodiments, a PSSR model as disclosed herein may be used forsubcellular structure segmentation (e.g., identifying cellularorganelles and the like), and to segment images with higher resolutionand capture finer details. In some embodiments, PSSR models as disclosedherein are configured to increase the SNR of an image by removing noisecomponents (e.g., “denoising”) and sources that may be trained into thePSSR model prior to image collection. In some embodiments, a PSSR modelas disclosed herein may be trained to correlate high-resolution imagescollected with one or more imaging techniques (e.g., light microscopyand EM). Accordingly, in some embodiments, a PSSR model may enhance anEM image of a region of a sample to a higher resolution based on acorresponding region of the sample captured by fluorescence microscopy.Moreover, in some embodiments, a PSSR model as disclosed herein may betrained to convolve images of the same region of a sample collected withdifferent techniques (and different resolution) into a single,comprehensive image of the region of the sample. Additionally, in someembodiments, the PSSR model may be able to deconvolve a high-resolutionimage into components corresponding to different scanning imagingtechniques (e.g., able to deconvolve to both EM and light microscopicimages from a single high-resolution image).

Embodiments as disclosed herein include PSSR models that are capable ofrestoring images under sampled by a factor, f, which can be 2, 3, oreven larger (e.g., 10, 16, 20, or more). Accordingly, a deeplearning-based image facilitates faster, lower dose imaging (e.g., lowerelectron radiation for SEM and lower photon dosage for scanning confocalmicroscopy) by about the same factor, f. This provides a technicaladvantage for the imaging system in terms of reduced sample damage, andreduced raw image file sizes that are buffered out of the system, e.g.,via a network link (wireless or wired). Accordingly, PSSR models asdisclosed herein provide a strategy for increasing the spatiotemporalresolution of point scanning imaging systems at the back end, topreviously unattainable levels, well beyond hardware limitations at thefront end (e.g., sample damage or imaging speed).

In embodiments including electron microscopy, methods and devices asdisclosed herein may provide significant advantages. Three-dimensionalelectron microscopy (3DEM) is a powerful technique for determining thevolumetric ultrastructure of tissues, which is desirable forconnectomics analysis of samples. In addition to serial section EM(ssEM) and focused ion beam SEM (FIB-SEM), one of the most common toolsfor high throughput 3DEM imaging is serial blockface scanning electronmicroscopy (SBFSEM), wherein a built-in ultramicrotome iteratively cutsultrathin sections (usually between 50-100 nm) off the surface of ablockface after it was imaged with a scanning electron probe. Thismethod facilitates relatively automated, high-throughput 3DEM imagingwith minimal post-acquisition image alignment. It is desirable to avoidhigher electron doses, as these cause sample charging, which renders thesample too soft to section and to image reliably. Furthermore,high-resolution 3DEM of relatively large volumes typically implies longimaging times and large file sizes and presents a significant bottleneckfor many labs. Thus, embodiments as disclosed herein include acquiring3DEM datasets with sub-Nyquist sampling (e.g., pixel sizes ≥4 nm), andpost-processing the resulting image to enhance the resolution to adesirable level using the model in the image processing engine.Increasing the resolution of the down sampled collection enables thereliable detection or analysis of smaller subcellular structures, suchas presynaptic vesicles. Accordingly, embodiments as disclosed hereincapture targeted regions of large (and maybe low-resolution) datasetsfor higher resolution ultrastructural information. Thus, the ability tocomputationally increase the resolution of these datasets is of highvalue to avoid sample damage and provide rapid image collection timesand protocols.

FIG. 1 illustrates an architecture 10 including imaging instrumentation130-1 (a light microscope), 130-2 (an electron/ion microscope), 130-3 (avideo camera), and 130-4 (a magnetic resonance imaging system—MRI—).Hereinafter, light microscope 130-1, electron/ion microscope 130-2,video camera 130-3, and MRI 130-4 will be collectively referred to as“imaging instrumentation 130”). As described herein, light microscope130-1 may include any instrument using optical imaging technology, suchas laser scanned fluorescence, confocal fluorescence, Ramanspectroscopy, atomic spectroscopy, laser induced fluorescence, and thelike. Electron/ion microscope 130-2 may include a SEM, a TEM, a STEM, oran ion-based imaging device. Imaging instrumentation 130 includeshardware such as optical components, electron guns, and other electroniccomponents and detectors to manipulate the hardware and interact with asample to generate imaging data. Accordingly, in some embodiments,imaging instrumentation 130 includes a point-scanning system configuredto form an image of a sample region by collecting data point by datapoint for a plurality of points in a region of the sample. Imaginginstrumentation 130 performs a point scanning of the image region basedon scanning parameters, including spatial resolution (or “slide”),radiation dosage, and dwell time. The resolution is the distance betweentwo neighboring data points in the sample. The radiation dosage is theamount of energy flux transmitted to each point in the sample, forimaging. The dwell time is the amount of time spent on each data point,including the delivering of the radiation, and the collection of ascattered radiation from the sample.

Architecture 10 also includes one or more associated control devices 110for enhanced imaging, according to various embodiments. Control device110 may include a computer, a display for displaying an output image,and input devices such as a keyboard, a mouse, and a stylus or pointer.Control device 110 may be configured to provide commands andinstructions to manipulate the hardware in imaging instrumentation 130.Imaging instrumentation 130 provides the imaging data from the sample tocontrol device 110. The data is transmitted to control device 110 togenerate an image on display 116. The data may also be stored in amemory of computer 111. The memory of computer 111 may also include aPSSR model having instructions which, when executed by computer 111,cause the computer to process the data from imaging instrumentation 130to generate an image in display 116.

In some embodiments, the PSSR model may be used by computer 111 toperform data analysis in real time, as imaging instrumentation 130 iscollecting an image, and provide instructions and commands, accordingly.For example, in some embodiments, the PSSR model may be configured toidentify a blood vessel or an intracellular structure as an image isbeing collected, and computer 111 may direct imaging instrumentation 130to scan the sample along the predicted path of the blood vessel, or tofocus and zoom-in on the intracellular structure. Furthermore, in someembodiments, the PSSR model may determine, based on a plurality ofcollected data points but before finalizing a scan of a portion of asample region, a change in one or more scanning parameters, based on apredicted image quality of the portion of the sample region.

In some embodiments, architecture 10 may include a network 150 to whichcontrol device 110 and imaging instrumentation 130 may becommunicatively coupled. In some embodiments, control device and imaginginstrumentation 130 may be remotely located from one another, and mayuse network 150 to communicate with each other. In that regard, network150 may be a private network, or a public network (such as the worldwide web), and may be implemented as a local area network (LAN) or awide area network (WLAN). Further, control device 110 and imaginginstrumentation 130 may communicate with network 150 via a wiredcommunication channel (e.g., telephone, Ethernet, or cable) or awireless communication channel (e.g., via a Wi-Fi or Bluetoothconnection, and the like). In some embodiments, control device 110 mayinclude input devices 114 (e.g., a mouse, a keyboard, and the like).

FIG. 2 illustrates a detailed block diagram of a system 200 inarchitecture 10, according to various embodiments. System 200 includescontrol device 110 and imaging instrumentation 130 (cf. FIG. 1)communicatively coupled with each other via external interfaces 218-1and 218-2 (hereinafter, collectively referred to as “external interfaces218”), respectively. External interfaces 218 may include wirelesscommunication components, e.g., radio-frequency (RF) antennas andradios, or wired components (e.g., copper wires, fiber optics, and thelike). Accordingly, external interfaces 218 may include hardware such asRF circuits, amplifiers, modulators and filters, and also software suchas digital processing software and the like. External interfaces 218 mayalso be configured to communicate with network 150 via a wired or awireless communication protocol (cf. FIG. 1). In some embodiments, asdescribed above, control device 110 and imaging instrumentation 130 maybe remotely located from one another, and external interfaces 218 maycommunicate with one another via network 150.

Imaging instrumentation may include a memory 220-2 storing instructions,and a processor 212-2 configured to execute the instructions and causeimaging instrumentation to perform at least one or more of the steps inmethods consistent with the present disclosure. For example, processor212-2 may cause imaging instrumentation 130 to collect an image from asample 250. In that regard, the instructions in memory 220-2 may beincluded in an application 242 for image collection and processing, andin a hardware driver 248, configured to provide commands to an imaginghardware 246. Imaging hardware includes electronic, mechanical, andoptical components that enable a probe 271 and a detector 272 tointeract with sample 250. In some embodiments, probe 271 is configuredto deliver a radiation to a selected point in sample 250, and detector272 is configured to measure a scattered radiation resulting from theinteraction between the radiation and sample 250.

In some embodiments, control device 110 includes a memory 220-1 storinginstructions, and a processor 212-1. Memory 220-1 may include an imageprocessing engine 222 and a hardware controller 228. Image processingengine 222 may include a model 224 (e.g., a PSSR model) and asegmentation tool 226. In some embodiments, model 224 may includedeconvolution algorithms such as structured illumination microscopy,single-molecule localization microscopy (SMLM), and pixel reassignmentmicroscopy. In some embodiments, image processing engine 222 mayconfigure imaging instrumentation 130 according to a pre-selectedpost-processing strategy. The power of image processing engine 222presents a new opportunity for redesigning imaging instrumentation 130and substantially reduces costs while extracting meaningful imagingdata. Similarly, image processing engine 222 may be configured toperform automated, real-time corrections to the images and real-timefeedback to the imaging hardware 246.

Hardware controller 228 is configured to provide instructions to andreceive status information from hardware driver 248 via externalinterfaces 218. In some embodiments, hardware controller 228communicates with image processing engine 222 to provide data from animage collection scan from imaging instrumentation 130. In someembodiments, hardware controller 228 may receive instructions from imageprocessing engine 222 to adjust or modify scanning parameters forimaging instrumentation 130. Processors 212-1 and 212-2 will becollectively referred to, hereinafter, as “processors 212.” Likewise,memories 220-1 and 220-2 will be collectively referred to, hereinafter,as “memories 220.”

One or more of processors 212 is configured to execute the instructionsand to cause system 200 to direct probe 271 to a first selected pointwithin a region of sample 250. One or more of processors 212 also causessystem 200 to select a radiation rate for probe 271 based on a desiredradiation dosage, to provide the radiation at the first selected point,based on the radiation rate, to collect at least a portion of thescattered radiation from the first selected point with detector 272, andto associate the first selected point with the portion of the scatteredradiation to form a first datum.

In some embodiments, input image 301 may include an array of pixelvalues. These pixel values, depending on the image resolution and size,may be an array of numbers corresponding to (length)×(width)×(number ofchannels). The number of channels can also be referred to as the‘depth.’ For example, the array could be L×W×Red Green Blue color model(RGB values). The RGB would be considered three channels, each channelrepresenting one of the three colors in the RGB color model. In someembodiments, system 200 may characterize a 20×20 image with arepresentative array of 20×20×3 (for RGB), with each point in the arrayassigned a value (e.g., 0 to 255) representing pixel intensity. A datummay include the point of the array and the assigned value or values(e.g., RGB values) for that point. Given this array of values, imageprocessing engine 222 obtains numbers indicating a probability of theimage being a certain class (e.g., 0.80 for ‘cell,’ 0.15 for ‘cellwall,’ and 0.05 for ‘no cell,’ or ‘interstitial’).

One or more of processors 212 also causes system 200 to direct probe 271to a second selected point within the region of sample 250 to form asecond datum based on a down sampling scheme stored in memory 220-1. Insome embodiments, the down sampling scheme is provided by model 224,which is trained with previously collected images stored in a trainingdatabase 252-1. More generally, model 224 may include any algorithm(linear and non-linear) trained to provide a simple answer based on acomplex input. Accordingly, the simple answer may be the down samplingscheme, and the complex input of multiple data points collected fromsample 250 during a current scanning session, or a previous scanningsession, or from a previously collected image. In that regard, model 224may include an artificial intelligence algorithm, a machine learningalgorithm, a deep learning algorithm, a neural network (NN), aconvolutional neural network (CNN, U-Net, and the like), a generativeadversarial neural network (GAN), a residual neural network (ResNet), orany combination of the above.

In some embodiments, training database 252-1 includes images from alarger universe of images stored in an image database 252-2. Memory220-1 may also store an annotated training database 252-3, whichincludes captions and other textual descriptions of the images stored intraining database 252-1. Annotated training database 252-3 may includean image classification value associated with each image in trainingdatabase 252-1. An image classification value may include a class or aprobability of classes that best describes an image (e. g., as in acaption, wherein each word or phrase has a specific code or value).Accordingly, and with the training from databases 252-1 and 252-2, imageprocessing engine 222 may be configured to identify objects of interestwithin an image in a database or provided by imaging instrumentation130, with a high level of accuracy using model 224, and segmentationtool 226.

In some embodiments, there is a direct, one-to-one correspondencebetween images in training database 252-1 and entries in annotatedtraining database 252-3. In some embodiments, training database 252-1and annotated training database 252-3 are embedded into one trainingdatabase. In addition to image database 252-2, or in conjunction withit, some embodiments may include an interaction history database 252-4.Interaction history database 252-4 may include, in addition topreviously collected images, metadata associated with the collection ofthe images. The metadata may include the technique used for imagecollection (e.g., SEM, STEM, confocal fluorescence, magnetic resonance,PET, and the like), scanning parameters (e.g., pixel resolution, dwelltime, radiation dosage, and the like), image classification values anddescriptors, and sample-related information (e.g., biological orphysiological descriptions of the region of the sample being imaged).Hereinafter, image database 252-2 and interaction history database 252-4will be collectively referred to as “databases 252.” In that regard,images and data from databases 252 may be collected by different imaginginstrumentation, and from different samples, at different locations andtimes. In some embodiments, at least one of databases 252 may beremotely located from control device 110. In such case, control device110 may access one or more of databases 252 through external interface218-1, through network 150. Accordingly, model 224 may be trained usingtraining data collected from any one of databases 252, training database252-1, and annotated training database 252-3, to better predict an imagequality from data being collected by imaging instrumentation 130 in realtime, or after a full scan of sample 250 (or a region thereof) has beencompleted.

One or more of processors 212 also causes system 200 to interpolate thefirst datum and the second datum based on a reverse of the down samplingscheme to obtain at least a third datum. In some embodiments, the thirddatum (e.g., the interpolation of the first datum and the second datum)may be provided by model 224 or by segmentation tool 226. Further, oneor more of processors 212 also causes system 200 to repeat the precedingsteps to cover a plurality of data from multiple selected points in aportion of the region of sample 250, and to form an image of the regionof the sample with the plurality of data, using image processing engine222. Control device 110 may also include an input device 214 and anoutput device 216 (e.g., input devices 114 and display 116, cf. FIG. 1).

FIG. 3 illustrates a block diagram of a convolutional neural network(CNN) 300 for enhanced imaging, according to various embodiments. Insome embodiments, CNN 300 may be included in an algorithm, or may becombined with a segmentation tool, as described above (e.g., model 224and segmentation tool 226, cf. FIG. 2). In some embodiments, CNN 300takes an input image 301 having a first resolution (e.g., number ofpixels, quality, and SNR), and generates an output image 302 having asecond resolution. More specifically, in some embodiments, CNN 300 isconfigured so that the second resolution is higher than the firstresolution (e.g., by a factor of less than 10, a factor of 10, or evenhigher). The image quality may be a single value indicative of imageresolution, SNR, and other values generally associated with imagequality such as sharpness, contrast, color range, blur, and the like. Insome embodiments, the image quality of output image 302 is higher thanthe image quality of input image 301 by a factor of less than 10, afactor of 10, or more.

In some embodiments, CNN 300 may include a single-frame neural network.In some embodiments, CNN 300 may include a ResNet-based U-Net fortraining (e.g., training model 224, cf. FIG. 2) and/or a GAN.

FIG. 4 illustrates a detailed block diagram of a CNN 400 for enhancedimaging, according to various embodiments. In some embodiments, CNN 400generally accomplishes an advanced form of image processing andclassification/detection by first looking for low level features suchas, for example, edges and curves, and then advancing to more abstract(e.g., unique to the type of images being classified) concepts through aseries of convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6,and 410-7 (hereinafter, collectively referred to as “convolutionallayers 410”). CNN 400 passes an input image 401 through a series ofconvolutional, non-linear, pooling (or down sampling, as will bediscussed in more detail below), and fully connected layers (e.g.,layers 410), and get an output 402. In some embodiments, output 402 maybe a single class or a probability of classes that best describes theimage or detects objects on the image, or a full image having a higher(or different) pixel count, a higher resolution, or a higher quality. Insome embodiments, CNN 400 includes a stack of distinct layers 410 thattransform input image 401 into output image 402 (e.g., holding the classscores) through a differentiable function. Input image 401 may be avolume (e.g., a three-dimensional—3D—object) wherein each datum is a‘voxel.’ Likewise, output 402 may be a volume formed by a plurality ofvoxels.

In some embodiments, layer 410-1 may be a convolutional layer (Conv)configured to process representative arrays of input image 401 using aseries of parameters. Rather than processing input image 401 as a whole,CNN 400 analyzes a collection of image sub-sets 412 using a filter (or‘neuron’ or ‘kernel’). Sub-sets 412 (or ‘regions’) may include a focalpoint in the array, as well as surrounding points. For example, a filtercan examine a series of 2×2 areas (or regions) in a 33×33 image. Regions412 can be referred to as receptive fields. Since the filter generallywill possess the same depth as the input, an image with dimensions of33×33×7 would have a filter of the same depth (e.g., 2×2×7). The actualstep of convolving, using the exemplary dimensions above, would involvesliding the filter along the input image, multiplying filter values withthe original pixel values of the image to compute element wisemultiplications, and summing these values to arrive at a single numberfor that examined region of the image.

After completion of this convolving step, using a 2×2×2 filter, anactivation map (or filter map) having dimensions of 31×31×5 will result.For each additional layer used, spatial dimensions are better preservedsuch that using 32 filters will result in 32 layers 410-1 (genericallyreferred to as ‘activation maps 410’) of 31×31×5 pixels. Each filterwill generally have a unique feature it represents (e.g., colors, edges,curves, and the like) that, together, represent the feature identifiersrequired for the final image output. These filters, when used incombination, allow CNN 400 to process input image 401 to detect thosefeatures present at each pixel. Therefore, if a filter serves as a curvedetector, the convolving of the filter along input image 401 produces anarray of numbers in the activation map that correspond to ‘highlikelihood’ of a curve (high summed element wise multiplications), ‘lowlikelihood’ of a curve (low summed element wise multiplications), or azero value at points where input image 401 includes no curve. As such,the greater number of filters (also referred to as channels) in theConv, the more depth (or data) that is provided on activation map 410,and therefore more information about the input that will lead to a moreaccurate output. In some embodiments, the outputs of layers 410 aretiled so that their input regions overlap, to obtain a betterrepresentation of input image 401. Tiling of layers 410 may be repeatedfor every such layer to allow CNNs to tolerate translation of the inputimage 401.

Balanced with accuracy of CNN 400 is the processing time and powerneeded to produce a result. In other words, the more filters (orchannels) used, the more time and processing power needed to execute theConv. Therefore, the choice and number of filters (or channels) to meetthe desired image enhancement in output 402 may be selected in view ofthe computational time and power available. To further enable a CNN todetect more complex features, additional Convs can be added to analyzeoutputs from the previous Conv (e.g., activation maps 410). For example,if a first Conv looks for a basic feature such as a curve or an edge, asecond Conv (e.g., layer 410-2, 410-3 through 410-7) can look for a morecomplex feature such as shapes, which can be a combination of individualfeatures detected in an earlier Conv layer (e.g., 410-1, 410-2 through410-7). By providing a series of Convs, CNN 400 can detect increasinglyhigher-level features to eventually arrive at a probability of detectinga complex object (e.g., mitochondria, a blood vessel, a plant root, aneuronal dendrite, and the like). Moreover, as the Convs stack on top ofeach other, analyzing the previous activation map output 410, each Convin the stack is naturally going to analyze a larger and larger receptivefield, by virtue of the scaling down that occurs at each Conv level,thereby allowing CNN 400 to respond to a growing region of pixel spacein detecting the object of interest.

CNN 400 includes a group of processing blocks, including at least oneprocessing block for convoluting input image 401 and at least one fordeconvolution (or transpose convolution). Additionally, the processingblocks can include at least one pooling block in convolutional layer410-4 and unpooling block 420 (e.g., pooling layers 314 and up samplinglayers 312, cf. FIG. 3). Pooling block 410-4 can be used to scale downan image in resolution to produce an output available for Conv. This canprovide computational efficiency (e.g., less time and less power usage),which can in turn improve actual performance of CNN 400. Pooling, or subsampling, enables blocks to keep filters small and computationalrequirements reasonable. In some embodiments, pooling block 410-4 maycoarsen output 402 (e.g., reducing spatial information within areceptive field), reducing it from the size of the input by a specificfactor. Unpooling blocks 420 can be used to reconstruct these coarseoutputs to produce output 402 with the same dimensions as input 401.Unpooling block 420 may include a reverse operation of a convolutingblock to return an activation output to the original input volumedimension.

However, the unpooling process generally just simply enlarges the coarseoutputs into a sparse activation map. To avoid this result, thedeconvolution block densifies this sparse activation map to produce bothan enlarged and dense activation map that eventually, after any furthernecessary processing, output 402 has size and density much closer toinput 401. As a reverse operation of the convolution block, rather thanreducing multiple array points in the receptive field to a singlenumber, the deconvolution block associates a single activation outputpoint with multiple outputs to enlarge and densify the resultingactivation output.

It should be noted that while pooling blocks can be used to scale downan image and unpooling blocks can be used to enlarge these scaled downactivation maps, convolution and deconvolution blocks can be structuredto both convolve/deconvolve and scale down/enlarge without the need forseparate pooling and unpooling blocks.

A processing block can include other layers that are packaged with aconvolutional or deconvolutional layer. These can include, for example,a ReLU or exponential linear unit layer (ELU), which are activationfunctions that examine the output from a Conv in its processing block.The ReLU or ELU layer acts as a gating function to advance only thosevalues corresponding to positive detection of the feature of interestunique to the Conv.

According to embodiments as disclosed herein, CNN 400 avoids the lostspatial information from pooling layer 410-4 and reduces/minimizesinternal covariate shifts inherent in a back-propagation process.Further, CNN 400 reduces processing time between input image 401 andoutput 402, which is desirable to achieve more complex feature detectionin image processing engines as disclosed herein.

In some embodiments, CNN 400 includes combinations of convolutional andfully connected layers, with pointwise nonlinearity applied at the endof or after each layer. A convolution operation on small regions ofinput image 401 may be introduced to reduce the number of freeparameters and improve generalization. In some embodiments, layers 410in CNN 400 may share weight parameters. Accordingly, the same filter(weights bank) may be used for each pixel in the layer, which reducesmemory footprint and improves performance.

In some embodiments, training of CNN 400 includes applying a progressiveresizing technique. The progressive resizing technique includes tworounds of training with HR images scaled to xy pixel sizes of 256×256and 512×512 and LR images scaled to 64×64 and 128×128 progressively(e.g., for SEM or STEM imaging instrumentation). The first round isinitiated with a pretrained ResU-Net, and CNN 400 trained from the firstround served as the pre-trained model for the second round. Theintuition behind this is it quickly reduces the training loss byallowing the model to see many images at a small scale during the earlystages of training. As the training progresses, CNN 400 focuses more onpicking up high-frequency features reflected through fine details thatare only stored within larger scale images. Therefore, features that arescale-variant can be recognized through the progressively resizedlearning at each scale.

Testing images for training CNN 400 may be cropped into smaller tilesbefore being fed into the model due to the memory limit of graphiccards.

FIG. 5 illustrates a block diagram of a generative adversarial network(GAN) 500 for enhanced imaging, according to various embodiments. GAN500 includes a generator network 501 competing with a discriminatornetwork 502. Generator network 501 adjusts its coefficients to createhigh-resolution synthetic images, while discriminator network 502adjusts its coefficients to distinguish the synthetic images created bygenerator network 501 versus high-resolution images originally obtainedwith an imaging instrumentation (e.g., imaging instrumentation 130, cf.FIGS. 1 and 2). In some embodiments, GAN 500 may be referred to as asuper-resolution GAN (SRGAN). The architecture of GAN 500 withcorresponding kernel size (k), number of feature maps (n), and stride(s) is indicated in the figure for each convolutional layer.

FIGS. 6A and 6B illustrate an image enhancement of a neuron obtained,according to various embodiments. The images shown may be collectedusing an imaging instrumentation communicatively coupled with a controldevice, as disclosed herein (e.g., control device 110 and imaginginstrumentation 130, cf. FIGS. 1 and 2). The imaging instrumentation isa confocal microscope, and the sample is a neuronal network.

FIG. 6A illustrates a low-resolution (LR) RGB image (225*150 pixels)600A generated by a 4× manual down sampling of a ‘ground truth’ image(GT). The GT image may be a ‘gold standard,’ high-resolution (HR) imageobtained using the same imaging instrumentation or the same technique.In some embodiments, the GT image may be obtained using a differentimaging instrumentation, or even a different technique (e.g., SEM, STEM,and the like).

FIG. 6B illustrates an up sampled high-resolution (HR) RGB image 600Bgenerated by a model in an image processing engine in the control device(e.g., model 224, cf. FIG. 2) when LR image 600A is provided as theinput (900×600 pixels). Images 600A and 600B will be collectivelyreferred to, hereinafter, as “images 600.”

FIGS. 7A through 7D illustrate image enhancements on a portion of theimages 600 obtained according to various embodiments. Images 700A, 700B,700C, and 700D will be collectively referred to, hereinafter, as “images700.” Images 700 show detailed comparisons of the same region croppedfrom images 600. The image enhancement in images 700 may be obtainedwith an image processing engine and a model in the control device (e.g.,image processing engine 222 and model 224, cf. FIG. 2).

FIG. 7A illustrates image 700A, which is the region cropped from LRimage 600A (30*20 pixels).

FIG. 7B illustrates image 700B, which is a cropped region of the GTimage corresponding to images 600. Image 700B has a notably higherresolution (120*80 pixels) than image 700A.

FIG. 7C illustrates image 700C, which is the same cropped region from HRimage 600B, when the model used to obtain the HR is bicubic (resolution:120*80 pixels).

FIG. 7D illustrates image 700D, which is the same cropped region from HRimage 600B, when the model used to obtain the HR is AI.

FIGS. 8A and 8B illustrate an image enhancement of a mitochondriaobtained according to various embodiments (images 800A and 800B,hereinafter, collectively referred to as “images 800”). Images 800 maybe collected using an imaging instrumentation communicatively coupledwith a control device, as disclosed herein (e.g., control device 110 andimaging instrumentation 130, cf. FIGS. 1 and 2). The imaginginstrumentation is a confocal microscope, and the sample is amitochondrion. The image enhancement in images 800 may be obtained withan image processing engine and a model in the control device (e.g.,image processing engine 222 and model 224, cf. FIG. 2).

FIG. 8A illustrates an LR grayscale image 800A (506*506 pixels)generated by a 4× manual down sampling of the GT image.

FIG. 8B illustrates an HR grayscale image 800B generated by the model inthe image processing engine using an AI algorithm when LR 800A is givenas the input (2024*2024 pixels).

FIGS. 9A through 9D illustrate an image enhancement of a portion ofimages 800, according to various embodiments (images 900A, 900B, 900C,and 900D, collectively referred to, hereinafter, as “images 900”).Images 900 show a detailed comparison of the same region cropped fromimages 800. The image enhancement in images 900 may be obtained with animage processing engine and a model in the control device (e.g., imageprocessing engine 222 and model 224, cf. FIG. 2).

FIG. 9A illustrates image 900A, which is the region cropped directlyfrom LR 800A (30*20 pixels).

FIG. 9B illustrates image 900B, which is the same region cropped fromthe GT (120*80 pixels).

FIG. 9C illustrates image 900C, which is the same region cropped from HR800B when the HR is generated by the model in the image processingengine using a bicubic algorithm (120*80 pixels).

FIG. 9D illustrates image 900D, which is the same region cropped from HR800B when the HR is generated by the model in the image processingengine using an AI algorithm (120*80 pixels).

FIGS. 10A through 10C illustrate an image enhancement of a sub-Nyquistunder sampled image obtained with a scanning electron microscope,according to various embodiments (images 1000A, 1000B, and 1000C,hereinafter, collectively referred to as “images 1000”). Images 1000 maybe collected using an imaging instrumentation communicatively coupledwith a control device, as disclosed herein (e.g., control device 110 andimaging instrumentation 130, cf. FIGS. 1 and 2). In some embodiments,the imaging instrumentation used to collect at least some of images 1000is SEM. More specifically, images 1000 are related to a same 70 nmsection of a myelin sheath in a neuronal network using SBFSEMinstrumentation and techniques. Image enhancement in images 1000 may beobtained with an image processing engine and a model in the controldevice (e.g., image processing engine 222 and model 224, cf. FIG. 2).Images 1000A and 1000C were acquired using a pixel size of 8 nm on abackscatter detector at 1000 Volts (e.g., 1 kV) and a current of221×10⁻¹² Amperes (e.g., 12 pico-A, or 12 pA). The pixel dwell time was2×10⁻⁶ seconds (e.g., 12 μs with an aperture of 30×10⁻⁶ meters) (e.g.,30 micro-meters, or 30 μm) and a working distance of 6.81 millimeters(mm). The section thickness was 100×10⁻⁹ meters (e.g., 100 nano-metersnm) and the field of view was 24.5×24.5 μm.

FIG. 10A illustrates LR image 1000A, including a cropped region obtainedwith a 4× under sampled imaging instrumentation.

FIG. 10B illustrates HR image 1000B of the same cropped region as image1000A, obtained with an AI model on LR image 1000A.

FIG. 10C illustrates GT image 1000C of the same cropped region as images1000A and 1000B, obtained with a high-resolution imaginginstrumentation. As can be seen, the quality of images 1000B and 1000Cis comparable, the difference being that the collection of data forobtaining image 1000B (e.g., the scanning of image 1000A) is lessinvasive, less costly, and much faster than the collection of GT image1000C.

FIG. 11 illustrates a detailed block diagram for a semi-synthetictraining scheme 1100 (referred to, hereinafter, as “scheme 1100”) in aneural network for enhanced imaging, according to various embodiments.Scheme 1100 may be applied to imaging instrumentation communicativelycoupled with a control device via external links and a network, whereinthe control device includes an image enhancement engine using a model(e.g., control device 110, external interfaces 218, image processingengine 222, model 224, network 150, and imaging instrumentation 130, cf.FIGS. 1 and 2). In some embodiments, semi-synthetic training scheme 1100may be applied for training the model to be applied on specificapplications, such as SBFSEM. More generally, semi-synthetic trainingschemes consistent with the present disclosure may be a semi-synthetictraining scheme 1100 and applied to any other imaging instrumentation(e.g., imaging instrumentation 130, cf. FIG. 1). In some embodiments, itis desired to obtain a high resolution (e.g., 2 nm) and SNR necessary toreliably detect presynaptic vesicles. In some embodiments, the model mayinclude a deep CNN (e.g., CNN 300 or CNN 400, cf. FIGS. 3 and 4) trainedon 2 nm high-resolution (HR) images. Accordingly, in some embodiments,scheme 1100 may be used to train the model to “super-resolve” 8 nmlow-resolution (LR) images to 2 nm resolution (PSSR model).

Scheme 1100 includes collecting a ˜130 giga-byte (GB) dataset of 2 nmpixel STEM images of 40 nm ultrathin sections from the hippocampus of amale rat. The collection of the HR images may be performed over thenetwork, using one of the external interfaces. To generatesemi-synthetic training pairs, a down sampling scheme (e.g., 16× downsampling and the like) is applied to the HR images. In some embodiments,the LR image may be simply a manually acquired LR image (such as with arapid scanning, low dosage point scan imaging instrumentation). In someembodiments, a semi-synthetic LR image obtained with down samplingschemes as disclosed herein may have a lower image quality than amanually acquired LR image. In some embodiments, scheme 1100 involvestraining a separate neural network to generate LR images. Alternatively,Gaussian and/or Poisson noise and blur may be added to the images inaddition to pixel downsampling to generate LR images. Each of the HRimages is paired with its corresponding LR image obtained from the downsampling scheme to form an HR-LR image pair. Scheme 1300 then trains theHR-LR image pairs on a ResNet-based U-Net model (e.g., CNN as disclosedherein, cf. FIGS. 3 and 4). In some embodiments, scheme 1300 may includea mean squared error (MSE) loss function that may produce visuallyimproved results, with high Peak SNR (PSNR) and positive StructuralSimilarity (SSIM) measurements.

Scheme 1100 includes evaluation metrics to determine image quality. Afirst metric may include a PSNR metric. In some embodiments, scheme 1100also uses an SSIM metric. PSNR and SSIM provide pixel-level datafidelity and perceptual quality fidelity correspondingly. The specificselection of a quality metric to use in scheme 1100 is non-limiting ofembodiments consistent with the present disclosure. PSNR is negativelycorrelated with MSE, numerically reflecting the pixel intensitydifference between the reconstruction image and the ground truth image,but it is also famous for poor performance when it comes to estimatinghuman perceptual quality. Instead of traditional error summationmethods, SSIM is designed to consider distortion factors like Luminancedistortion, contrast distortion, and loss of correlation wheninterpreting image quality.

Scheme 1100 incudes collecting real-world LR images with the same pixeldwell time as the HR data, resulting in 16× lower signal for thereal-world LR images (e.g., because there are 16× fewer pixels). In someembodiments, HR-STEM training images may have a higher image qualitythan the HR validation dataset acquired on an SEM (e.g., for obtainingthe CNN model that is to be trained with scheme 1100). Accordingly, insome embodiments, scheme 1300 may produce a model that is trained torestore an LR collected image into an HR synthetic image having a higherimage quality than an HR collected image using the same imaginginstrumentation. Moreover, scheme 1100 also provides a model that may beused with a wider range of real world data, including data extractedfrom a database, through the network, and which was collected from adifferent imaging instrumentation at a different location, through thenetwork.

Low-resolution (LR) images were generated from high-resolution (HR) EMor confocal images using an image baselining function 1110. Due to thevariance of image format, image size, dynamic range, depth, and the likein the acquired high-resolution images, data cleaning is desirable togenerate training sets that can be easily accessed during training.Consistent with this disclosure, “data sources” may refer to uncleanedimages acquired with high-resolution imaging instrumentation, while“data sets” may refer to images generated and preprocessed from “datasources.” In addition to baselining function 1110, some embodimentsinclude a data augmentation tool such as random cropping, dihedralaffine function, rotation, and random zoom to increase the variety andsize of the training data.

FIGS. 12A through 12C illustrate some applications of a neural networkfor electron microscopy, according to various embodiments (images 1200Aand 1200B, collectively referred to, hereinafter, as “images 1200”).Some of images 1200 may be collected using an imaging instrumentationcommunicatively coupled with a control device, as disclosed herein(e.g., control device 110 and imaging instrumentation 130, cf. FIGS. 1and 2). Some of images 1200 may be obtained through a model in an imageprocessing engine, as disclosed herein (e.g., image processing engine222 and model 224, cf. FIG. 2). In some embodiments, the model may befurther trained with a semi-synthetic training scheme, as disclosedherein (e.g., scheme 1100, cf. FIG. 11). Images 1200 may be obtained bythe controller device via external links and a network (e.g., externalinterfaces 218 and network 150, cf. FIGS. 1 and 2). Images 1200correspond to STEM images of 40 nm ultrathin sections from thehippocampus of a male rat. In some embodiments and without limitation,some of the images may correspond to 80 nm sections imaged with abackscatter detector. Without limiting the scope of embodimentsdisclosed herein, and for illustrative purposes only, the HR in images1200 corresponds to a 2 nm resolution, and in some embodiments, the LRcorresponds to an 8 nm resolution.

FIG. 12A illustrates images 1200A including LR image 1200A-1, asynthetic HR image 1200A-2 (wherein the up sampling algorithm includes abilinear interpolation), a synthetic HR image 1200A-3 (wherein the upsampling algorithm includes a PSSR model), and a collected HR image1200A-4 (e.g., imaging instrumentation operating in high-resolutionmode, or GT image). The PSSR-restored images from the semi-syntheticpairs contained more detail and yet displayed less noise, making iteasier to discern fine details such as presynaptic vesicles.

FIG. 12B illustrates images 1200B including LR image 1200B-1, asynthetic HR image 1200B-2 (wherein the up sampling algorithm includes abilinear interpolation), a synthetic HR image 1200B-3 (wherein the upsampling algorithm includes a PSSR model), and a collected HR image1200B-4 (e.g., imaging instrumentation operating in high-resolutionmode, or GT image). The PSSR-restored images from the semi-syntheticpairs contained more detail and yet displayed less noise, making iteasier to discern fine details such as presynaptic vesicles. The secondrow of images 1200B is the zoomed in image corresponding to the croppedregion, as indicated in the first row of FIG. 1200B. A further zoom inview of a crop region is indicated in the inset for images 1200B in thesecond row.

FIG. 12C illustrates charts 1200C with tests of the image qualityobtained after enhancing LR images with either a bilinear interpolationmodel and a PSSR model for both STEM instrumentation (graphs 1210-1) andSEM instrumentation (graphs 1210-2). Graphs 1210-1 and 1210-2 will becollectively referred to, hereinafter, as “graphs 1210.” Each data pointin graphs 1210 is associated to an image pair, indicating at least oneLR image collection and one HR image collection (to use as a GT point ofcomparison). Thus, in some embodiments, the multiple images used to formgraphs 1210 may be retrieved from multiple instruments in multipleremote locations, through the network. Graphs 1210 include two differentimage quality metrics: PSNR and SSIM, which illustrate the robustness ofdeep learning-based image restoration models as disclosed herein withrespect to variations in image properties, notwithstanding the use of amodel generated from training images acquired in one condition on imagesacquired in another (e.g., data generated using a different samplepreparation technique, type, or on a different microscope). PSSRsignificantly increases both the resolution and quality (PSNR, SSIM) ofmanually acquired low-resolution images. Thus, PSSR models as disclosedherein may be applied to data acquired by different instruments, andeven with different techniques and samples.

Mouse FIB-SEM data sample preparation and image acquisition settingswere previously described in the original manuscript the datasets werepublished. Briefly, the images were acquired with 4 nm voxel resolution.We down sampled the lateral resolution to 8 nm, and then applied thePSSR model to the down sampled data to ensure the proper 8-to-2 nmtransformation for which the PSSR was trained.

The rat SEM data sample was acquired from an 8-week old male Wistar ratthat was anesthetized with an overdose of pentobarbital (75 mg kg-1) andperfused through the heart with 5-10 ml of a solution of 250 mM sucrose5 mM MgCl₂ in 0.02 M phosphate buffer (pH 7.4) (PB) followed by 200 mlof 4% paraformaldehyde containing 0.2% picric acid and 1% glutaraldehydein 0.1 M PB. Brains were then removed and oblique horizontal sections(50 μm thick) of frontal cortex/striatum were cut on a vibratingmicrotome along the line of the rhinal fissure. The tissue was stainedand cut to 50 nm sections using ATUMtome for SEM imaging. The rat SEMdata was acquired using an acceleration voltage of 1.5 kV and a dwelltime of 3 μs, using the backscatter detector with a pixel resolution of10×10 nm (which was up sampled to 8 nm using bilinear interpolation).

FIGS. 13A through 13E illustrate some applications of a neural networkfor electron microscopy, according to various embodiments. Images 1300A,1300B, 1300C, 1300D, and 1300E (hereinafter, collectively referred to as“images 1300”) may be collected using an imaging instrumentationcommunicatively coupled with a control device, as disclosed herein(e.g., control device 110 and imaging instrumentation 130, cf. FIGS. 1and 2). Some of images 1300 may be obtained through a model in an imageprocessing engine, as disclosed herein (e.g., image processing engine222 and model 224, cf. FIG. 2). In some embodiments, the model may befurther trained with a semi-synthetic training scheme, as disclosedherein (e.g., scheme 1300, cf. FIG. 13). Images 1300 may be obtained bythe controller device via external links and a network (e.g., externalinterfaces 218 and network 150, cf. FIGS. 1 and 2). Images 1300correspond to brain tissue (neuronal networks and synaptic interfaces).In some embodiments and without limitation, some of the images maycorrespond to mouse, rat, and fly samples imaged on four differentmicroscopes in four different data sources (e.g., laboratories). Withoutlimiting the scope of embodiments disclosed herein, and for illustrativepurposes only, the HR in images 1300 corresponds to a 2 nm resolution,and in some embodiments, the LR corresponds to an 8 nm resolution.

In some embodiments (e.g., fly brain tissue), at least some of images1300 include FIB-SEM imaging instrumentation. For this, images may beacquired with 10 nm voxel resolution and up sampled to 8 nm usingbilinear interpolation. A PSSR model is applied to the up sampled datato obtain an 8 nm-to-2 nm transformation (for which the PSSR model wastrained). In addition to the SBFSEM and FE-SEM imaging systems, PSSRprocessing appeared to restore images 1300 from multiple data sources.In some embodiments, a further segmentation step using a segmentationtool (e.g., segmentation tool 226, cf. FIG. 2) after applying imageenhancement with the PSSR model is substantially simplified, and can beperformed automatically, without human intervention, according to someembodiments. A segmentation step is desirable for analyzing 3DEMdatasets. Remarkably, the PSSR model also performed well on a 10×10×10nm resolution FIB-SEM fly brain dataset, resulting in a 2×2×10 nmresolution dataset with higher SNR and resolution. Thus, a PSSR model asdisclosed herein may be used for a down sampling factor of 25×, or evenhigher, increasing the lateral resolution and speed of FIB-SEM imagingby a factor of up to 25×, or maybe more.

FIG. 13A illustrates images 1300A from a portion of a mouse braintissue. Images 1300A include LR image 1300A-1, a synthetic HR image1300A-2 (wherein the up sampling algorithm includes a bilinearinterpolation), and a synthetic HR image 1300A-3 (wherein the upsampling algorithm includes a PSSR model). The PSSR-restored images fromthe semi-synthetic pairs contained more detail and yet displayed lessnoise, making it easier to discern fine details such as presynapticvesicles. The second row of images 1300A is the zoomed in imagecorresponding to the cropped region, as indicated in the first row ofFIG. 1300A. A further zoom in view of a crop region is indicated in theinset for images 1300A in the second row. Images 1300A show that thePSSR model is able to restore an 8 nm pixel SBFSEM 3D dataset to 2 nm.

FIG. 13B illustrates images 1300B from a portion of a fly brain tissue.Images 1300B include LR image 1300B-1, a synthetic HR image 1300B-2(wherein the up sampling algorithm includes a bilinear interpolation),and a synthetic HR image 1300B-3 (wherein the up sampling algorithmincludes a PSSR model). The PSSR-restored images from the semi-syntheticpairs contained more detail and yet displayed less noise, making iteasier to discern fine details such as presynaptic vesicles. The secondrow of images 1300B is the zoomed in image corresponding to the croppedregion, as indicated in the first row of FIG. 1300B. A further zoom inview of a crop region is indicated in the inset for images 1300B in thesecond row.

FIG. 13C illustrates images 1300C from a mouse brain tissue. Images1300C include LR image 1300C-1, a synthetic HR image 1300C-2 (whereinthe up sampling algorithm includes a bilinear interpolation), and asynthetic HR image 1300C-3 (wherein the up sampling algorithm includes aPSSR model). The PSSR-restored images from the semi-synthetic pairscontained more detail and yet displayed less noise, making it easier todiscern fine details such as presynaptic vesicles. The second row ofimages 1300C is the zoomed in image corresponding to the cropped region,as indicated in the first row of FIG. 1300C. A further zoom in view of acrop region is indicated in the inset for images 1300C in the secondrow.

FIG. 13D illustrates images 1300D from a rat brain tissue. Images 1300Dinclude LR image 1300D-1, a synthetic HR image 1300D-2 (wherein the upsampling algorithm includes a bilinear interpolation), and a syntheticHR image 1300D-3 (wherein the up sampling algorithm includes a PSSRmodel). The PSSR-restored images from the semi-synthetic pairs containedmore detail and yet displayed less noise, making it easier to discernfine details such as presynaptic vesicles. The second row of images1300D is the zoomed in image corresponding to the cropped region, asindicated in the first row of FIG. 1300D. A further zoom in view of acrop region is indicated in the inset for images 1300D in the secondrow.

FIG. 13E illustrates images 1300E-1, 1300E-2, 1300E-3, and 1300E-4(hereinafter, collectively referred to as “images 1300E”) from a ratbrain tissue. Images 1300E include LR image 1300E-1, a synthetic HRimage 1300E-2 (wherein the up sampling algorithm includes a bilinearinterpolation), a synthetic HR image 1300E-3 (wherein the up samplingalgorithm includes a PSSR model), and a collected HR image 1300E-4(e.g., imaging instrumentation operating in high-resolution mode, or GTimage). The PSSR-restored images from the semi-synthetic pairs containedmore detail and yet displayed less noise, making it easier to discernfine details such as presynaptic vesicles.

Graphs 1310-1, 1310-2, and 1310-3 (hereinafter, collectively referred toas “graphs 1310”) indicate an image processing accuracy, with emphasison the possibility of false positives (e.g., artifacts or“hallucinations”). Some of images 1300E (e.g., HR synthetic images1300E-3) include 2 nm pixel SBFSEM datasets which may be beyond thecapabilities for at least some imaging instrumentation used. In someembodiments, this may preclude the generation of GT validation imagesfor the SBFSEM data (e.g., GT images 1300E-4). To identify a trust levelof processed datasets for which no GT data exists, some embodimentsinclude determining whether the PSSR model is sufficiently accurate foruseful downstream analysis.

To do this, at least some of images 1300E include low 8 nm and high 2 nmpixel resolution SEM image pairs of ultrathin sections. Further, someimages 1300E include 16× super-sampled of the 8 nm pixel images (LR) to2 nm pixel images (HR) using either bilinear interpolation or a PSSRmodel. The image quality (PSNR and/or SSIM) of LR-bilinear and LR-PSSRmodels for the above two procedures should be equal in the absence ofartifacts (slope=1 in graphs 1310). Comparing the image quality of theresulting HR synthetic images 1300E-4 for the two cases, it is seen thatLR-PSSR (cf. graph 1310-2) significantly outperforms LR-bilinear (cf.graph 1310-1). To further test the accuracy and utility of the PSSRoutput in a more concrete, biological context, graph 1310-3 illustratesa randomized LR-bilinear, LR-PSSR, and HR images, wherein two blindedhuman experts perform manual segmentation of presynaptic vesicles (onehuman for each of the two up sampling procedures). Graphs 1310illustrate that LR-PSSR is significantly more accurate than theLR-bilinear, and even the two human experts. In some embodiments, theLR-PSSR output reduced false negatives by ˜300%, and the LR-PSSR outputmay have a slightly higher number of “false positives” than theLR-bilinear. However, since the HR data is noisier than both thetraining data as well as the LR-PSSR output, it is possible that not allof the false positives are truly false. Moreover, the variance betweenthe LR-PSSR and HR results was similar to the variance between the twoexpert human results on HR data. The human results in fact may be a goldstandard (near maximum accuracy and precision possible). Graphs 1310reveal that in some embodiments, PSSR models as disclosed herein mayeffectively produce 2 nm 3DEM data from 8 nm resolution acquisitions,revealing important subcellular structures, otherwise lost in many 3DEMdatasets. Furthermore, the ability to reliably 16× super-sample lowerresolution datasets presents an opportunity to increase the throughputof SEM imaging by at least one order of magnitude, according to someembodiments.

FIGS. 14A through 14K illustrate some applications in confocalfluorescence microscopy, according to various embodiments. Images 1400A,1400B, 1400C, 1400D, 1400E, and 1400F (hereinafter, collectivelyreferred to as “images 1400”) may be collected using an imaginginstrumentation communicatively coupled with a control device, asdisclosed herein (e.g., control device 110 and imaging instrumentation130, cf. FIGS. 1 and 2). The imaging instrumentation used may include alaser scanning confocal microscope. Some of images 1400 may be obtainedthrough a model in an image processing engine, as disclosed herein(e.g., image processing engine 222 and model 224, cf. FIG. 2). In someembodiments, the model may be further trained with a semi-synthetictraining scheme, as disclosed herein. Images 1400 may be obtained by thecontroller device via external links and a network (e.g., external links218 and network 150, cf. FIGS. 1 and 2). For example, in someembodiments, a ˜5 GB of high-resolution confocal time lapses ofmitochondrial transport in neurons may be either collected or downloadedfrom the network. Images 1400 show that a PSSR model as disclosed hereinmay up sample and restore under sampled confocal images obtained withstandard imaging instrumentation (e.g., using a single photomultipliertube—PMT—detector), to an HR image having an image quality comparable tothe GT (e.g., gold standard in imaging instrumentation). Using thefine-tuned PSSR model, images 1400 include restored HR images from downto 16× lower resolution. Accordingly, in some embodiments, scheme 1600may provide a PSSR model enabling 16× faster time lapses of neuronalmitochondria motility to sufficient resolution for proper analysis.Without limiting the scope of embodiments disclosed herein, and forillustrative purposes only, some of the HR images 1400 and the LR images1400 have similar resolution of laser scanning confocal images asdisclosed herein.

FIG. 14A illustrates images 1400A including LR image 1400A-1, asynthetic HR image 1400A-2 (wherein the up sampling algorithm includes abilinear interpolation), a synthetic HR image 1400A-3 (wherein the upsampling algorithm includes a PSSR model), and a collected HR image1400A-4 (e.g., imaging instrumentation operating in high-resolutionmode, or GT image). Moreover, images 1400A may be split intosemi-synthetic rows 1401 (downloaded images from the network, orsynthetically down sampled HR images), and real-world rows 1402 (imagescollected with imaging instrumentation).

FIG. 14B illustrates images 1400B including LR image 1400B-1, asynthetic HR image 1400B-2 (wherein the up sampling algorithm includes abilinear interpolation), and a synthetic HR image 1400B-3 (wherein theup sampling algorithm includes a PSSR model). The improved speed,resolution, and SNR observed in image 1400B-3 enables detection ofmitochondrial traffic event 1411 that were not detectable in LR image1400B-1.

FIG. 14C illustrates images 1400C including LR image 1400C-1, and asynthetic HR image 1400C-2 (wherein the up sampling algorithm includes aPSSR model). Images 1400C illustrate a number of mitochondria within aneuronal cytoplasm. Images 1401-1 and 1401-2 (collectively referred to,hereinafter, as “streak images 1401”) illustrate a time-lapse (with timeproceeding downward) streak of the two-dimensional images 1400C,respectively (1801-1 for the LR-confocal collection, and 1401-2 for thesynthetic HR up sampling image using the PSSR model). A zoomed-inportion illustrates some detail in streak images 1401. Traces 1412 instreak images 1401 indicate a moving mitochondrion, and the slope oftraces 1412 indicates the speed. Clearly, the SNR in image 1400C-2 andstreak image 1401-2 is substantially higher than in the corresponding LRsource (e.g., image 1400C-1 and streak image 1401-1).

FIG. 14D illustrates images 1400D including LR image 1400D-1, and asynthetic HR image 1400D-2 (wherein the up sampling algorithm includes aPSSR model). Images 1400D illustrate still images of a number ofmitochondria within a neuronal cytoplasm, at different moments in time.Images 1400D illustrate that because the collection of synthetic HRimage 1400D-2 is 16× faster than HR-confocal imaging, instantaneousmotility details were preserved (e.g., better time resolution), whereasin LR image 1400D-1 they were lost.

FIG. 14E illustrates images 1400E including LR image 1400E-1, asynthetic HR image 1400E-2 (wherein the up sampling algorithm includes atransfer-learning PSSR model, hereinafter referred to as “PSSR-TLmodel”), and a collected HR image 1400E-3 (e.g., imaging instrumentationoperating in high-resolution mode, or GT image). The PSSR-TL modelincludes a pretrained PSSR model modified to use features of previouslystored PSSR models for similar, or related applications (e.g., retrievedfrom annotated training database 252-3 or interaction history database252-4). Images 1400E demonstrate that a PSSR-TL model as disclosedherein significantly improves the image quality compared to the originalLR-image 1400E-1, and has even higher SNR than GT image 1400E-3. Theimage enhancement provided by a PSSR-TL model as disclosed hereinenables resolution of two mitochondria moving past one another in theaxon, as well as fission and fusion events. By comparing the richness ofcrossing traces in synthetic HR image 1400E-2 with the fainter and fewercrossing traces in GT image 1400E-3, it is clear that the PSSR-TL modelis able to catch motility phenomena that even the GT technique misses.

FIG. 14F illustrates images 1400F including LR image 1400E-1, asynthetic HR image 1400E-2 (wherein the up sampling algorithm includes abilinear interpolation), and a synthetic HR image 1400E-3 (wherein theup sampling algorithm includes a PSSR-TL model). The improved speed,resolution, and SNR observed in images 1400E-3, enable detection ofmitochondrial traffic events 1411 that were not detectable in the raw LRimages. Each row of images 1400F corresponds to a different time duringimage collection.

FIG. 14G illustrates image quality graphs 1410-1 and 1410-2(hereinafter, collectively referred to as “graphs 1410”) for multipleimages 1400C after applying a PSSR-TL model for resolution enhancement.Graphs 1410-1 correspond to semi-synthetic pair 1401, and graphs 1410-2correspond to real-world pairs 1402, both using the two image qualitymeasures as disclosed herein (PSNR and SSIM). The improvement in imagequality introduced by PSSR-TL is evident.

FIGS. 14H-14I illustrate charts 1420 h and 1420 i (hereinafter,collectively referred to as “charts 1420”), including a statisticalanalysis of mitochondrial motility results obtained from images 1400C,1400D, and 1400E (cf. FIGS. 14C through 14E). Charts 1420 are separatedby column according to whether the data pictured is collected from an LRimage (“LR”), a synthetic HR image (“HR”), or a GT image (“GT”). Chart1420 h illustrates a histogram of total distance traveled by themitochondria (μm), and chart 1420 i illustrates a histogram of themeasured velocity of the mitochondria (μm/s). Notably, the overall totaldistance mitochondria travelled in axons was the same for bothLR-confocal and HR-confocal. In addition, the synthetic HR imageindicates a much-varied distribution of mitochondrial velocities (e.g.,a larger range), while the GT image shows almost no ability to measurevelocity (with an average very close to zero).

FIGS. 14J through 14K illustrate charts 1430 j and 1430 k (hereinafter,collectively referred to as “charts 1430”), including a statisticalanalysis of mitochondrial motility results obtained from images 1400C,1400D, and 1400E (cf. FIGS. 14C through 14E). Charts 1430 are separatedby column according to whether the data pictured is collected from an LRimage (“LR”), a synthetic HR image (“HR”), or a GT image (“GT”). Chart1430 j illustrates a histogram of the time spent in motion by themitochondria (in percent, %), and chart 1430 k illustrates acomplementary histogram of the time spent in the stopped position by themitochondria (in percent, %). Small distances travelled were easy todefine in the LR-PSSR-TL images, and therefore there was an overallreduction in the percent of time mitochondria spent in the stoppedposition in the LR-PSSR-TL data. Overall, LR-PSSR-TL and HR-confocalprovided similar values for the percent time mitochondria spent inmotion.

FIG. 15 is a flow chart illustrating steps in a method for controllingan imaging instrumentation to obtain an enhanced image, according tovarious embodiments. Method 1500 may be performed by one or more controldevices coupled with image instrumentation (e.g., control device 130 andimage instrumentation 110, cf. FIGS. 1 and 2). For example, at leastsome of the steps in method 1500 may be performed by one component in acomputer in the control device or an application in the imageinstrumentation (e.g., external interfaces 218, application 242,hardware driver 248, imaging hardware 246, probe 271 and detector 272,cf. FIG. 2). Accordingly, at least some of the steps in method 1500 maybe performed by a processor executing commands stored in a memory of oneor more computers and devices as disclosed herein (e.g., processors 212and memories 220, cf. FIG. 2). The memory may also include a trainingdatabase, an annotated training database, an image database, or aninteraction history database storing data and information that may beused in one or more steps in method 1500 (e.g., databases 252, cf. FIG.2). In some embodiments, the control device, the imaginginstrumentation, and the database in method 1500 may be communicativelycoupled via a network (e.g., network 150, cf. FIG. 1). Further, invarious embodiments, at least some of the steps in method 1500 may beperformed overlapping in time, almost simultaneously, or in a differentorder from the order illustrated in method 1500. Moreover, a methodconsistent with various embodiments disclosed herein may include atleast one, but not all, of the steps in method 1500.

Step 1502 includes selecting a radiation level for a first probe to meeta desired radiation dosage.

In some embodiments, step 1502 includes preparing the sample prior toplacing it in the imaging instrumentation. For example, in someembodiments, step 1502 may include growing cells in DMEM supplementedwith 10% fetal bovine serum at 37° C. with 5% CO2. In some embodiments,step 1502 may include plating cells onto either 8-well #1.5 imagingchambers or #1.5 35 mm dishes (Cellvis), and coating the dishes with 10μg/mL fibronectin in PBS at 37° C. for 30 minutes prior to plating. Insome embodiments, step 1502 may include adding 50 nM MitoTracker DeepRed or CMXRos Red for 30 minutes, and washing the dish for at least 30minutes to allow for recovery time before imaging in FluoroBrite media.

In some embodiments, step 1502 may include preparing a neuronal tissuesample. For example, in some embodiments, step 1502 includes preparingprimary hippocampal neurons prepared from E18 rat (Envigo) embryos aspreviously described. In some embodiments, step 1502 includes dissectinghippocampal tissue from embryonic brain and further dissociating to asingle hippocampal neuron by trypsinization with Papain. In someembodiments, step 1502 includes plating the prepared neurons oncoverslips coated with 3.33 μg/mL laminin and 20 μg/mL poly-L-Lysine ata density of 7.5×104 cells/cm². In some embodiments, step 1502 includesmaintaining the cells in Neurobasal medium supplemented with B27,penicillin/streptomycin, and L-glutamine for 7-21 days in vitro. In someembodiments, step 1502 includes transfecting the hippocampal neurons twodays before imaging, with Lipofectamine 2000.

Step 1504 includes providing, with the first probe, a first radiationamount at a first selected point within a region of the sample, based onthe radiation level.

Step 1506 associating the first selected point with at least the portionof a first emitted radiation resulting from an interaction of the firstradiation amount with the sample, to form a first datum.

Step 1508 includes identifying a second selected point within the regionof the sample based on a down sampling scheme.

Step 1510 includes providing, with the first probe, a second radiationamount at the second selected point within the region of the sample.

Step 1512 associating the second selected point with at least theportion of a second emitted radiation resulting from the interaction ofthe second radiation amount with the sample, to form a second datum.

Step 1514 includes interpolating the first datum and the second datumbased on an up sampling scheme to obtain at least a third datum.

Step 1516 includes obtaining a plurality of data from multiple selectedpoints in a portion of the region of the sample.

Step 1518 includes forming an image of the region of the sample with theplurality of data.

In some embodiments, step 1518 includes segmenting the image. Forexample, in some embodiments, step 1518 includes aligning rigid imagesets generated from the same region of neuropil (LR-Bilinear; LR-PSSR;HR), identifying, and cropping presynaptic axonal boutons (n=10) fromthe image set. In some embodiments, step 1518 includes assigningrandomly generated file names to the image sets from the threeconditions, and distributing to two blinded human experts for manualcounting of presynaptic vesicles. In some embodiments, step 1518includes identifying vesicles by identifying a clear and completemembrane, being round in shape, and of approximately 35 nm in diameter.In some embodiments, step 1518 includes de-selecting, for consistencybetween human experts, vesicles that were embedded in or attached toobliquely sectioned, axonal membranes. In some embodiments, step 1518includes counting docked and non-docked synaptic vesicles as separatepools. In some embodiments, step 1518 includes recording, unblinding,and grouping vesicle counts, by condition and by expert counter. In someembodiments, step 1518 includes conducting a linear regression analysisbetween the counts of the HR images and the corresponding images of thetwo different LR conditions (LR-bilinear; LR-PSSR), and determining howclosely the counts corresponded between the HR and LR conditions. Insome embodiments, step 1518 includes conducting a linear regressionanalysis to determine the variability between counters.

FIG. 16 is a flow chart illustrating steps in a method for training analgorithm to control imaging instrumentation to obtain an enhancedimage, according to various embodiments. Method 1600 may be performed byone or more control devices coupled with image instrumentation (e.g.,control device 130 and image instrumentation 110, cf. FIGS. 1 and 2).For example, at least some of the steps in method 1600 may be performedby one component in a computer in the control device or an applicationin the image instrumentation (e.g., external interfaces 218, hardwaredriver 248, and application 242, cf. FIG. 2). Accordingly, at least someof the steps in method 1600 may be performed by a processor executingcommands stored in a memory of one or more computers and devices asdisclosed herein (e.g., processors 212 and memories 220, cf. FIG. 2).The memory may also include a training database, an annotated trainingdatabase, an image database, or an interaction history database storingdata and information that may be used in one or more steps in method1600 (e.g., databases 252, cf. FIG. 2). In some embodiments, the controldevice, the imaging instrumentation, and the database in method 1600 maybe communicatively coupled via a network (e.g., network 150, cf. FIG.1). Further, in various embodiments, at least some of the steps inmethod 1600 may be performed overlapping in time, almost simultaneously,or in a different order from the order illustrated in method 1600.Moreover, a method consistent with various embodiments disclosed hereinmay include at least one, but not all, of the steps in method 1600.

Step 1602 includes retrieving a high-resolution image of a known sample.

In some embodiments, step 1602 includes collecting a confocal image ofU2OS cells with a 63×1.4NA oil objective on a confocal system with aninverted stage and heated incubation system with 5% CO₂ control. In someembodiments, step 1602 includes directing, for both HR and LR images, alaser power of 2.5 μW and a pixel dwell time of 1.15 μs/pixel. In someembodiments, step 1602 includes acquiring HR-confocal images with a 2×Nyquist pixel size of 49 nm/pixel in SR mode (e.g., a virtual pinholesize of 2.5 AU), and processing the images using auto-filter settings.In some embodiments, step 1602 includes acquiring low-resolutionconfocal images (LR-confocal) with the same settings but with 0.5×Nyquist pixel size (196 nm/pixel).

In some embodiments, step 1602 includes retrieving neuronal mitochondriaimaging and kymograph analysis. For example, in some embodiments, step1602 includes imaging live-cell primary neurons using a confocalmicroscope enclosed in a temperature control chamber at 37° C. and 5%CO₂, using a 63× (NA 1.4) oil objective in SR-confocal mode (e.g., 2.5AU virtual pinhole). In some embodiments, step 1602 includes, for LRimaging, acquiring images with a confocal PMT detector having a pinholesize of 2.5 AU at 440×440 pixels at 0.5× Nyquist (170 nm/pixel) every270.49 ms using a pixel dwell time of 1.2 μs and a laser power rangingbetween 1-20 μW. In some embodiments, step 1602 includes acquiring HRimages at 1764×1764 pixels at 2× Nyquist (42.5 nm/pixel) every 4.33 susing a pixel dwell time of 1.2 μs and a laser power of 20 μW. In someembodiments, step 1602 includes collecting imaging data using software,and processing HR images using a confocal processing. In someembodiments, step 1602 includes analyzing time-lapse moviesautomatically.

In some embodiments, step 1602 includes retrieving HR STEM data. Forthis, in some embodiments, step 1602 includes preparing tissue from aperfused 7-month old male rat, cut from the left hemisphere, stratumradiatum of CA1 of the hippocampus. In some embodiments, step 1602includes staining, embedding, and sectioning the tissue at 45 nm, andimaging tissue sections with a STEM detector with a 28 kV acceleratingvoltage and an extractor current of 102 μA (gun aperture 30 μm). In someembodiments, step 1602 includes acquiring images with a 2 nm pixel sizeand a field size of 24576×24576 pixels, having a working distance fromthe specimen to the final lens as 3.7 mm, and the dwell time as 1.2 μs.

For the testing and validation ground truth data sets, step 1602 mayinclude acquiring paired LR and HR images of the adult mouse hippocampaldentate gyrus middle molecular layer neuropil from ultrathin sections(80 nm), collected on silicon chips and imaged in an SEM. In someembodiments, step 1602 includes collecting pairs of 4×4 μm images fromthe same region at pixel sizes of both 8 nm and 2 nm, and an SEM set at:3 kV; dwell time, 5.3 μs; line averaging, 2; aperture, 30 μm; workingdistance, 2 mm.

Step 1604 includes identifying a first classifier for thehigh-resolution image of the known sample, wherein the first classifierincludes a first value.

Step 1606 includes aggregating, with a selected coefficient, one or morepixels in the high-resolution image to obtain a low-resolution image ofthe sample, wherein the one or more pixels are selected based on adesired down sampling of an image collection system. In someembodiments, step 1606 includes image baselining. In some embodiments,step 1606 includes lowering the SNR of one or more pixel values in theHR image corresponding with the same field of view taken under theimaging instrumentation. In some embodiments, step 1606 includescollecting the HR image and the baseline image obtained in step 1606 toform an image pair for further training the algorithm. In someembodiments, step 1606 includes normalizing the high-resolution imagefrom 0 to 1 before aggregating the pixel values (e.g., a 1000×1000 pixelimage would be down sampled to 250×250 pixels in a 16× reductionfactor). In some embodiments, step 1606 includes rescaling theaggregated pixel value to 8-bit format [0 to 255] for viewing withnormal image analysis software. In some embodiments, step 1606 includesusing spline interpolation of order 1 or more, for aggregating the pixelvalues.

Step 1608 includes obtaining a second classifier for the low-resolutionimage of the sample, wherein the second classifier includes a secondvalue.

Step 1610 includes determining a metric value with a difference betweenthe second value and the first value. In some embodiments, step 1610includes selecting a metric threshold based on a desired image quality,and modifying the selected coefficient when a metric value surpasses themetric threshold. Step 1610 may further include storing the selectedcoefficient with a model for up sampling an image.

Computer System

FIG. 17 is a block diagram that illustrates a computer system 1700, uponwhich embodiments, or portions of the embodiments, of the presentteachings may be implemented. In various embodiments of the presentteachings, computer system 1700 can include a bus 1708 or othercommunication mechanism for communicating information, and a processor1702 coupled with bus 1708 for processing information. In variousembodiments, computer system 1700 can also include a memory 1704, whichcan be a random access memory (RAM) or other dynamic storage device,coupled to bus 1708 for determining instructions to be executed byprocessor 1702. Memory 1704 also can be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 1702. In various embodiments,computer system 1700 can further include a read-only memory (ROM) orother static data storage device 1706 coupled to bus 1708 for storingstatic information and instructions for processor 1702. A storage device1706, such as a magnetic disk or optical disk, can be provided andcoupled to bus 1708 for storing information and instructions.

In various embodiments, computer system 1700 can be coupled via bus 1708and input/output module 1710 to a display 1716, such as a cathode raytube (CRT) or liquid crystal display (LCD), for displaying informationto a computer user. An input device 1714, including alphanumeric andother keys, can be coupled to input/output module 1710 for communicatinginformation and command selections to processor 1702. Another type ofuser input device 1714 is a cursor control, such as a mouse, atrackball, or cursor direction keys for communicating directioninformation and command selections to processor 1702 and for controllingcursor movement on display 1716. This input device 1714 typically hastwo degrees of freedom in two axes, a first axis (e.g., x) and a secondaxis (e.g., y), that allows the device to specify positions in a plane.However, it should be understood that input devices 1714 allowing for3-dimensional (x, y, and z) cursor movement are also contemplatedherein. A communications module 1712 may also be coupled withinput/output module 1710, and configured to communicate with an externaldevice or network (e.g., via a modem, Ethernet card, Wi-Fi antenna, RFantenna, and the like).

Consistent with certain implementations of the present teachings,results can be provided by computer system 1700 in response to processor1702 executing one or more sequences of one or more instructionscontained in memory 1704. Such instructions can be read into memory 1704from another computer-readable medium or computer-readable storagemedium, such as storage device 1706. Execution of the sequences ofinstructions contained in memory 1704 can cause processor 1702 toperform the processes described herein. Alternatively, hard-wiredcircuitry can be used in place of or in combination with softwareinstructions to implement the present teachings. Thus, implementationsof the present teachings are not limited to any specific combination ofhardware circuitry and software.

The term “computer-readable medium” (e.g., data store, data storage,etc.) or “computer-readable storage medium” as used herein refers to anymedia that participates in providing instructions to processor 1702 forexecution. Such a medium can take many forms, including but not limitedto, non-volatile media, volatile media, and transmission media. Examplesof non-volatile media can include, but are not limited to, optical,solid state, and magnetic disks, such as data storage device 1706.Examples of volatile media can include, but are not limited to, dynamicmemory, such as memory 1704. Examples of transmission media can include,but are not limited to, coaxial cables, copper wire, and fiber optics,including the wires that include bus 1708.

Common forms of computer-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, a RAM, PROM, and EPROM, aFLASH-EPROM, any other memory chip or cartridge, or any other tangiblemedium from which a computer can be read.

In addition to a computer-readable medium, instructions or data can beprovided as signals on transmission media included in a communicationsapparatus or system to provide sequences of one or more instructions toprocessor 1702 of computer system 1700 for execution. For example, acommunication apparatus may include a transceiver having signalsindicative of instructions and data. The instructions and data areconfigured to cause one or more processors to implement the functionsoutlined in the disclosure herein. Representative examples of datacommunications transmission connections can include, but are not limitedto, telephone modem connections, wide area networks (WAN), local areanetworks (LAN), infrared data connections, NFC connections, etc.

It should be appreciated that the methodologies described hereinincluding flow charts, diagrams, and the accompanying disclosure can beimplemented using computer system 1700 as a standalone device or on adistributed network of shared computer processing resources such as acloud computing network.

In accordance with various embodiments, the systems and methodsdescribed herein can be implemented using computer system 1700 as astandalone device or on a distributed network of shared computerprocessing resources such as a cloud computing network. As such, anon-transitory computer-readable medium can be provided in which aprogram is stored for causing a computer to perform the disclosedmethods for identifying mutually incompatible gene pairs.

It should also be understood that the preceding embodiments could beprovided, in whole or in part, as a system of components integrated toperform the methods described. For example, in accordance with variousembodiments, the methods described herein can be provided as a system ofcomponents or stations for analytically determining novelty responses.

In describing the various embodiments, the specification may havepresented a method and/or process as a particular sequence of steps.However, to the extent that the method or process does not rely on theparticular order of steps set forth herein, the method or process shouldnot be limited to the particular sequence of steps described. As one ofordinary skill in the art would appreciate, other sequences of steps maybe possible. Therefore, the particular order of the steps set forth inthe specification should not be construed as limitations on the claims.In addition, the claims directed to the method and/or process should notbe limited to the performance of their steps in the order written, andone skilled in the art can readily appreciate that the sequences may bevaried and still remain within the spirit and scope of the variousembodiments. Similarly, any of the various system embodiments may havebeen presented as a group of particular components. However, thesesystems should not be limited to the particular set of components, theirspecific configuration, communication, and physical orientation withrespect to each other. One skilled in the art should readily appreciatethat these components can have various configurations and physicalorientations (e.g., wholly separate components, units, and subunits ofgroups of components, and different communication regimes betweencomponents).

Although specific embodiments and applications of the disclosure havebeen described in this specification (including the associatedAppendix), these embodiments and applications are exemplary only, andmany variations are possible.

Recitation of Embodiments

A. A method for collecting an image from a sample includes selecting aradiation level for a first probe to meet a desired radiation dosage.The method also includes providing, with the first probe, a firstradiation amount at a first selected point within a region of thesample, based on the radiation level, and associating the first selectedpoint with at least the portion of a first emitted radiation resultingfrom an interaction of the first radiation amount with the sample, toform a first datum. The method also includes identifying a secondselected point within the region of the sample based on a down samplingscheme, providing, with the first probe, a second radiation amount atthe second selected point within the region of the sample, andassociating the second selected point with at least the portion of asecond emitted radiation resulting from the interaction of the secondradiation amount with the sample, to form a second datum. The methodalso includes interpolating the first datum and the second datum basedon an up sampling scheme to obtain at least a third datum, obtaining aplurality of data from multiple selected points in a portion of theregion of the sample, and forming an image of the region of the samplewith the plurality of data.

B. A system for collecting an image from a sample, includes a firstprobe configured to deliver a radiation to a selected point in thesample, a first detector configured to measure a scattered radiationresulting from an interaction between the radiation and the sample, amemory storing instructions and one or more processors configured toexecute the instructions. When executing the instructions, the one ormore processors cause the system to select a radiation level for a firstprobe to meet a desired radiation dosage, to provide, with the firstprobe, a first radiation amount at a first selected point within aregion of the sample, based on the radiation level, and to associate thefirst selected point with at least the portion of a first emittedradiation resulting from an interaction of the first radiation amountwith the sample, to form a first datum. The one or more processors alsocause the system to identify a second selected point within the regionof the sample based on a down sampling scheme, to provide, with a firstprobe, a second radiation amount at the second selected point within theregion of the sample, and to associate the second selected point with atleast the portion of a second emitted radiation resulting from theinteraction of the second radiation amount with the sample, to form asecond datum. The one or more processors also cause the system tointerpolate the first datum and the second datum based on an up samplingscheme to obtain at least a third datum, to obtain a plurality of datafrom multiple selected points in a portion of the region of the sample,and to form an image of the region of the sample with the plurality ofdata.

C. A computer-implemented method to train an algorithm for collecting animage of a sample includes retrieving a high-resolution image of a knownsample and identifying a first classifier for the high-resolution imageof the known sample, wherein the first classifier includes a firstvalue. The computer implemented method also includes aggregating, with aselected coefficient, one or more pixels in the high-resolution image toobtain a low-resolution image of the sample, wherein the one or morepixels are selected based on a desired down sampling of an imagecollection system, and obtaining a second classifier for thelow-resolution image of the sample, wherein the second classifierincludes a second value. The computer-implemented method also includesdetermining a metric value with a difference between the second valueand the first value, modifying the selected coefficient when the metricvalue is at least equal to a selected metric threshold, and storing theselected coefficient with a model for up sampling an image obtained withthe image collection system when the metric value is smaller than theselected metric threshold.

Each of embodiments A, B or C may be combined with the followingelements in any number and order, to produce further embodimentsconsistent with the present disclosure, as follows:

Element 1, wherein selecting a radiation level for the first probefurther includes selecting a dwell time and a resolution, and modifyingat least one parameter from a group consisting of the dwell time, theresolution, and the radiation level, prior to forming the image of theregion of the sample to meet a threshold quality of the image of theregion of the sample. Element 2, wherein the first probe is an electrongun, and selecting a radiation level includes selecting a voltage and acurrent for the electron gun. Element 3, with the first probe, a secondradiation amount at the second selected point within the region of thesample includes selecting a point within a distance from the firstselected point to meet a minimum resolution allowed by the down samplingscheme and to meet a quality threshold value of the image of the regionof the sample. Element 4, wherein providing, with the first probe, asecond radiation amount at the second selected point within the regionof the sample includes increasing a radiation level in the secondradiation amount when a predicted quality of the image of the region ofthe sample is below a threshold quality value. Element 5, whereinproviding, with the first probe, a second radiation amount to a secondselected point within the region of the sample includes increasing aradiation level in the second radiation amount when a predicted qualityof the image of the region of the sample is above a threshold qualityvalue. Element 6, wherein the first probe is an electron gun, theplurality of data includes a scattered electron flux, the down samplingscheme is obtained from a training data collected from a confocalfluorescence image, and providing, with the first probe, a secondradiation amount to a second selected point includes selecting alocation of the second selected point based on a correlation between theconfocal fluorescence image with a location of first selected point.Element 7, wherein interpolating the first datum and the second datumincludes fitting a bilinear function to the first datum and the seconddatum. Element 8, wherein forming an image of the region of the samplewith the plurality of data includes modifying at least a location or asignal amplitude value from one of the plurality of data to meet aquality threshold value of the image of the region of the sample.Element 9, wherein forming an image of the region of the sample with theplurality of data includes modifying at least a location or a signalamplitude value from multiple data in the plurality of data to meet aquality threshold value of the image of the region of the sample.Element 10, wherein forming an image of the region of the sample with aplurality of data includes correlating at least some of the plurality ofdata with a training data set. Element 11, wherein forming an image ofthe region of the sample with a plurality of data includes matching,through a correlation, the image of the region of the sample with aregion of the sample from another image in a training data set. Element12, wherein forming an image of the region of the sample includesremoving a noise component from the plurality of data, wherein the noisecomponent is identified with a training data set that is used to selectthe down sampling scheme. Element 13, wherein forming an image of theregion of the sample includes identifying a noise source with a trainingdata set that is used to select the down sampling scheme, from a groupconsisting of a radiation level noise, a probe jitter noise, a detectornoise, a scattering noise, and a thermal noise. Element 14, wherein thefirst probe is a laser scanning first probe and the plurality of dataincludes confocal fluorescence data, and forming an image of the regionof the sample with a plurality of data includes convolving the pluralityof data with a training data set obtained with an electron microscope ata higher resolution. Element 15, further including reducing a distancebetween the second selected point and the first selected point when apredicted quality of the image of the region of the sample is below adesired quality threshold. Element 16, further including increasing adistance between the second selected point and the first selected pointwhen a predicted quality of the image of the region of the sample isabove a desired quality threshold. Element 17, the method furtherincluding determining a dwell time for collecting at least a portion ofthe first emitted radiation resulting from the interaction of the firstradiation amount with the sample. Element 18, wherein the secondselected point is approximately at a same location as the first selectedpoint, further including forming the second datum at a selected timeinterval after forming the first datum, the method further includingadding the image of the region of the sample to a time-lapse file.Element 19, further including selecting a time interval based on thedown sampling scheme, forming the second datum at the time intervalafter forming the first datum, and identifying a physiologicalphenomenon in the region of the sample based on a similarity of theplurality of data with the physiological phenomenon in a trainingtime-lapse file stored in a database. Element 20, further includingselecting a time interval to collect a time-lapse plurality of data,identifying a physiological phenomenon from a difference between theplurality of data and the time-lapse plurality of data. Element 21,further including grouping at least a portion of the data by identifyingan anatomic segment in the portion of the region of the sample, whereinthe anatomic segment of the sample is at least a portion of one of: ablood vessel, a plant root, a cellular organelle, a neuronal axon, abrain, a neuronal synapsis, a blood vessel, or a subcellular structure.Element 22, further including predicting an image quality value of theimage based on the up sampling scheme before forming the image of theregion of the sample with plurality of data, and reducing at least oneof the radiation level or an image resolution when the image qualityvalue of the image of the region is higher than a quality threshold.Element 23, wherein the sample is a living sample, the method furtherincluding collecting one or more images of the region of the sample atdifferent times, to form a time-lapse file. Element 24, wherein thesample is a living sample, the method further including identifying aphysiological phenomenon in the living sample when the up samplingscheme includes a plurality of data indicating a displacement of acomponent in the image of the region of the sample, and wherein thephysiological phenomenon includes at least one of a mitochondrialfission, a mitochondrial displacement or a vesicular transition across acellular membrane. Element 25, further including zooming in on a portionof the region of the sample and increasing an image resolution for theportion of the region of the sample. Element 26, further includingdetermining a physiological information about the sample from adisplacement in a component of the image of the region of the sample.Element 27, further including determining a physiological informationabout the sample with a statistical distribution of a velocity of acomponent of the image of the region of the sample. Element 28, furtherincluding determining multiple velocities of multiple components of theimage of the region of the sample and identifying a physiologicalinformation about the sample when the velocities of multiple componentsfit an expected pattern.

Element 29, wherein at least some of the instructions in the memoryinclude multiple coefficients in one of a neural network, a machinelearning algorithm, or an artificial intelligence algorithm. Element 30,wherein the first probe includes an electron gun, the radiation is anelectron flux, and the radiation level includes a voltage and a currentof the electron flux. Element 31, wherein the first probe is a laserbeam configured to operate in a continuous mode or in a pulsed modeaccording to the radiation level and a wavelength of the laser beam.Element 32, wherein the first probe includes a radiofrequency source,further including a first detector including a magnet configured tomeasure a decay of a resonant magnetization in the sample. Element 33,wherein the first probe includes a radioactive isotope embedded in thesample, further including a first detector including one of a gamma raydetector, a beta ray detector, or a positron detector. Element 34,further including a first detector including an anode configured todetect an electron beam scattered off a surface of the sample. Element35, further including a first detector including an anode configured todetect an electron beam scattered through a bulk of the sample. Element36, further including a first detector including an optical detectorconfigured to measure a scattered radiation having a wavelength selectedfrom a fluorescence spectrum of at least a portion of the sample.Element 37, further including a filter configured to separate the firstradiation amount from the first probe from the first emitted radiationresulting from the interaction of the first radiation amount with thesample.

Element 38, wherein aggregating one or more pixels in thehigh-resolution image to obtain a low-resolution image of the sampleincludes using a bilinear interpolation function between a value foreach of the one or more pixels in the high-resolution image. Element 39,wherein aggregating one or more pixels in the high-resolution imageincludes convolving the one or more pixels using the selectedcoefficient as a convolution factor. Element 40, wherein aggregating oneor more pixels in the high-resolution image includes randomly injectingGaussian additive noise to an aggregated pixel value. Element 41,wherein aggregating one or more pixels in the high-resolution imageincludes randomly injecting salt-and-pepper noise to an aggregated pixelvalue. Element 42, wherein aggregating one or more pixels in thehigh-resolution image includes increasing a signal-to-noise ratio of anaggregated pixel value. Element 43, wherein aggregating one or morepixels in the high-resolution image includes associating a baselineimage including an aggregated pixel with the high-resolution image toform an image pair for further training the algorithm. Element 44,wherein aggregating one or more pixels in the high-resolution imageincludes interpolating multiple pixel values with a spline function oforder of 1 or more. Element 45, wherein modifying the selectedcoefficient when the metric value is at least equal to a selected metricthreshold includes transferring a value from one or more coefficients inone or more algorithms associated with a similar image of the sampleinto the selected coefficient. Element 46, further including modifyingthe selected coefficient according to a learning rate and determiningthe learning rate from an amount of weight update per iteration toarrive at optimal value. Element 47, further including obtaining a thirdclassifier for the low-resolution image of the sample based on a secondselected coefficient, the method further including modifying the secondselected coefficient when the metric value is smaller than the selectedmetric threshold and storing the second selected coefficient when themetric value is at least equal to the selected metric threshold. Element48, further including storing the high-resolution image and thelow-resolution image, and the metric value as a training set. Element49, further including selecting a second image from a second knownsample, back-propagating the second image to obtain a back-propagatedimage, and modifying the selected coefficient when a second metric valueis lower than the selected metric threshold, the second metric valuebeing indicative of a difference between the back-propagated image andthe second image. Element 50, further including determining a lossfunction based on a difference between the image of the known sample anda backpropagation image obtained with a reverse algorithm that is aninverse of the algorithm including the selected coefficient, andmodifying the selected coefficient when the loss function has a valuelarger than a pre-selected threshold. Element 51, further includingperforming a backward pass to determine a contributing factor to a lossfunction that evaluates a difference between the image of a second knownsample and a backpropagation image obtained with the selectedcoefficient. Element 52, further including updating a filter coefficientto improve a convolutional neural network. Element 53, further includingselecting a lower learning rate and increasing the selected metricthreshold. Element 54, further including weighting an internal covariateshift in a back-propagation of a second image of a second known samplewhen updating the selected coefficient. Element 55, further includingselecting the metric value based on a peak signal-to-noise-ratio of thehigh-resolution image and a peak signal-to-noise ratio of a synthetichigh-resolution image obtained from the low-resolution image and thesecond classifier. Element 56, further including calculating the metricvalue based on a structural similarity between the high-resolution imageand a synthetic high-resolution image obtained from the low-resolutionimage and the second classifier.

1. A method for collecting an image from a sample, comprising: selectinga radiation level for a first probe to meet a desired radiation dosage;providing, with the first probe, a first radiation amount at a firstselected point within a region of the sample, based on the radiationlevel; associating the first selected point with at least the portion ofa first emitted radiation resulting from an interaction of the firstradiation amount with the sample, to form a first datum; identifying asecond selected point within the region of the sample based on a downsampling scheme; providing, with the first probe, a second radiationamount at the second selected point within the region of the sample;associating the second selected point with at least the portion of asecond emitted radiation resulting from the interaction of the secondradiation amount with the sample, to form a second datum; interpolatingthe first datum and the second datum based on an up sampling scheme toobtain at least a third datum; obtaining a plurality of data frommultiple selected points in a portion of the region of the sample; andforming an image of the region of the sample with the plurality of data.2. The method of claim 1, wherein selecting a radiation level for thefirst probe further comprises selecting a dwell time and a resolution,and modifying at least one parameter from a group consisting of thedwell time, the resolution, and the radiation level, prior to formingthe image of the region of the sample to meet a threshold quality of theimage of the region of the sample. 3-4. (canceled)
 5. The method ofclaim 1, wherein providing, with the first probe, a second radiationamount at the second selected point within the region of the samplecomprises either increasing a radiation level in the second radiationamount when a predicted quality of the image of the region of the sampleis below a threshold quality value, or decreasing a radiation level inthe second radiation amount when a predicted quality of the image of theregion of the sample is above a threshold quality value.
 6. (canceled)7. The method of claim 1, wherein the first probe is an electron gun,the plurality of data includes a scattered electron flux, the downsampling scheme is obtained from a training data collected from aconfocal fluorescence image, and providing, with the first probe, asecond radiation amount to a second selected point comprises selecting alocation of the second selected point based on a correlation between theconfocal fluorescence image with a location of first selected point. 8.(canceled)
 9. The method of claim 1, wherein forming an image of theregion of the sample with the plurality of data comprises modifying atleast a location or a signal amplitude value from at least one of theplurality of data to meet a quality threshold value of the image of theregion of the sample. 10-13. (canceled)
 14. The method of claim 1,wherein forming an image of the region of the sample comprisesidentifying a noise source with a training data set that is used toselect the down sampling scheme, from a group consisting of a radiationlevel noise, a probe jitter noise, a detector noise, a scattering noise,and a thermal noise.
 15. (canceled)
 16. The method of claim 1, furthercomprising reducing a distance between the second selected point and thefirst selected point when a predicted quality of the image of the regionof the sample is below a desired quality threshold, or increasing adistance between the second selected point and the first selected pointwhen a predicted quality of the image of the region of the sample isabove a desired quality threshold.
 17. (canceled)
 18. The method ofclaim 1, the method further comprising determining a dwell time forcollecting at least a portion of the first emitted radiation resultingfrom the interaction of the first radiation amount with the sample. 19.(canceled)
 20. The method of claim 1, further comprising selecting atime interval based on the down sampling scheme, forming the seconddatum at the time interval after forming the first datum, andidentifying a physiological phenomenon in the region of the sample basedon a similarity of the plurality of data with the physiologicalphenomenon in a training time-lapse file stored in a database. 21.(canceled)
 22. The method of claim 1, further comprising predicting animage quality value of the image based on the up sampling scheme beforeforming the image of the region of the sample with plurality of data,and reducing at least one of the radiation level or an image resolutionwhen the image quality value of the image of the region is higher than aquality threshold.
 23. (canceled)
 24. The method of claim 1, wherein thesample is a living sample, the method further comprising identifying aphysiological phenomenon in the living sample when the up samplingscheme includes a plurality of data indicating a displacement of acomponent in the image of the region of the sample, and wherein thephysiological phenomenon comprises at least one of a mitochondrialfission, a mitochondrial displacement or a vesicular transition across acellular membrane.
 25. (canceled)
 26. The method of claim 1, furthercomprising determining a physiological information about the sample froma displacement in a component of the image of the region of the sample,or with a statistical distribution of a velocity of a component of theimage of the region of the sample. 27-28. (canceled)
 29. A system forcollecting an image from a sample, comprising: a first probe configuredto deliver a radiation to a selected point in the sample; a firstdetector configured to measure a scattered radiation resulting from aninteraction between the radiation and the sample; a memory storinginstructions; and one or more processors configured to execute theinstructions and to cause the system to: select a radiation level for afirst probe to meet a desired radiation dosage; provide, with the firstprobe, a first radiation amount at a first selected point within aregion of the sample, based on the radiation level; associate the firstselected point with at least the portion of a first emitted radiationresulting from an interaction of the first radiation amount with thesample, to form a first datum; identify a second selected point withinthe region of the sample based on a down sampling scheme; provide, witha first probe, a second radiation amount at the second selected pointwithin the region of the sample; associate the second selected pointwith at least the portion of a second emitted radiation resulting fromthe interaction of the second radiation amount with the sample, to forma second datum; interpolate the first datum and the second datum basedon an up sampling scheme to obtain at least a third datum; obtain aplurality of data from multiple selected points in a portion of theregion of the sample; and form an image of the region of the sample withthe plurality of data. 30-32. (canceled)
 33. The system of claim 29,wherein the first probe comprises a radiofrequency source, furthercomprising a first detector including a magnet configured to measure adecay of a resonant magnetization in the sample.
 34. The system of claim29, wherein the first probe comprises a radioactive isotope embedded inthe sample, further comprising a first detector including one of a gammaray detector, a beta ray detector, or a positron detector.
 35. Thesystem of claim 29, further comprising a first detector including ananode configured to detect an electron beam scattered off a surface ofthe sample or through a bulk of the sample. 36-38. (canceled)
 39. Acomputer-implemented method to train an algorithm for collecting animage of a sample, comprising: retrieving a high-resolution image of aknown sample; identifying a first classifier for the high-resolutionimage of the known sample, wherein the first classifier includes a firstvalue; aggregating, with a selected coefficient, one or more pixels inthe high-resolution image to obtain a low-resolution image of thesample, wherein the one or more pixels are selected based on a desireddown sampling of an image collection system; obtaining a secondclassifier for the low-resolution image of the sample, wherein thesecond classifier includes a second value; determining a metric valuewith a difference between the second value and the first value;modifying the selected coefficient when the metric value is greater thana threshold; and storing the selected coefficient with a model for upsampling an image when the metric value is smaller than the threshold.40. (canceled)
 41. The computer-implemented method claim 39, whereinaggregating one or more pixels in the high-resolution image comprisesconvolving the one or more pixels using the selected coefficient as aconvolution factor.
 42. The computer-implemented method of claim 39,wherein aggregating one or more pixels in the high-resolution imagecomprises randomly injecting Gaussian additive noise, or salt-and-peppernoise, to an aggregated pixel value. 43.-58. (canceled)